The present invention relates generally to user input systems for computing devices and, more particularly, to an optical stylus for providing input.
User input may be provided to a computing device in a variety of different ways. Common examples include a keyboard, a mouse, a trackpad, a stylus, a camera, a microphone and the like. Some computing devices allow for user input via a display screen, such as for touch screen devices. The touch screens allow for a user to interact directly with displayed objects without the use of a cursor or other input device.
There are many different types of touch sensing technologies that may be implemented in a touch screen. In some embodiments, a capacitive touch screen may be implemented which relies on changes in capacitive measurements to determine when a user is providing input. One capacitive-type touch screen, in particular, may rely upon interaction with another capacitive member in order to sense contact with its surface as an input. As such, fingers may generally be used, but plastic devices (e.g., styluses) are ineffective to register input. Moreover, a capacitive member configured to register an input with these types of touch screens is the same size of a finger tip providing for imprecise input.
In some embodiments, a stylus is provided with an optical sensor, such as a camera, that is used in determining a location and movement of the stylus relative to a touch screen display of a computing device. It should be appreciated, however, that displays other than touch screens may be implemented in some embodiments. The optical stylus may be configured to transmit the location and movement to the computing device. In some embodiments, the optical stylus may be configured to process and/or filter the location and movement information prior to transmission, whereas in other embodiments, raw data may be transmitted.
In some embodiments, the relative position of the optical stylus may be determined based on indicia detectable by the optical stylus. The indicia may further be used in determining the movement of the optical stylus. The indicia may include pixel dependent indicia that are communicated via the pixels displayed by the touch screen or physical or permanent indicia that are physically present on or in the screen or otherwise positioned such that the optical stylus may detect them. Generally, the indicia are imperceptible to the human eye. As such, the touch screen may be encoded without diminishing or otherwise interfering with images displayed on the touch screen.
In some embodiments, the optical stylus may be configured to determine when the stylus is in contact with the touch screen. For example, in some embodiments, the stylus may include a pressure sensor that may be used to determine when the optical stylus is in contact with the touch screen. In some embodiments, the optical stylus may implement optical sensors to determine when contact is made.
The optical stylus may include one or more orientation determining sensors, such as accelerometers and/or gyroscopes. For example, the optical stylus may include an accelerometer. The accelerometer may be configured to aid in movement determinations, as well as a orientation. In particular, the accelerometer may allow for a determination to be made as to a particular angle at which the stylus is being held relative to the surface of the touch screen.
While multiple embodiments are disclosed, still other embodiments of the present invention will become apparent to those skilled in the art from the following Detailed Description. As will be realized, the embodiments are capable of modifications in various aspects, all without departing from the spirit and scope of the embodiments. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not restrictive.
Embodiments discussed herein may take the form of, or operate with, an optical stylus that allows for position-based sense writing on a touch screen display. The contact of the optical stylus with the touch screen may be differentiated from input from a hand or finger. The optical stylus works in addition to capacitive sensing, may provided sub-millimeter scale resolution and, in some embodiments, may have a pointed tip to provide precision input. Additionally, the stylus operates independent of an orientation or rotation of the stylus and/or screen. In some embodiments, the stylus may be pressure sensitive and may communicate data to a host device wirelessly.
The optical stylus is provided with an optical sensor or camera that may determine a relative location, angle and/or movement of the stylus with respect to a display a computing device. In some embodiments, the display may be a touch screen display, such as a capacitive touch screen. The optical stylus may be configured to transmit the location, angle and movement to the computing device. In some embodiments, the optical stylus may be configured to process and/or filter the location and movement information prior to transmission, whereas in other embodiments, raw data may be transmitted.
The relative position of the optical stylus may be determined based on indicia detectable by the optical stylus. The indicia may further be used in determining the movement of the optical stylus. These indicia may be, for example, pixel-dependent patterns that are graphically presented on the touch screen. Alternatively, the indicia may be physical or permanent markings or patterns that are incorporated into the screen, or are otherwise positioned such that the optical stylus may detect them. Generally, the indicia are imperceptible to the human eye. As such, the touch screen may be encoded without diminishing or otherwise interfering with images displayed on the touch screen.
In some embodiments, the optical stylus may be configured to determine when the stylus is in contact with the touch screen. For example, in some embodiments, the stylus may include a pressure sensor that may be used to determine when the optical stylus is in contact with the touch screen. In some embodiments, the optical stylus may implement optical sensors to determine when contact is made. That is, in some embodiments, the camera may be used to determine if contact has been made with an encoded display.
The optical stylus may include one or more orientation sensors, such as accelerometers and/or gyroscopes. For example, the optical stylus may include an accelerometer. The accelerometer may be configured to aid in movement determinations, as well as in determining orientation. In particular, the accelerometer may allow for a determination to be made as to a particular angle at which the stylus is being held relative to the surface of the touch screen. Moreover, in some embodiments, the orientation of the stylus relative to a display may be determined, at least in part, using information collected by the camera, such as using the aspect ratio of the patterns or markings being imaged.
Turning now to the drawings and referring initially to
A pressure sensor 22 may be located under, adjacent to or coupled with the end cover 16, in some embodiments. In particular, as illustrated, in some embodiments, the pressure sensor 22 may be located between the end cover 16 and the body 12 of the stylus 10. In other embodiments, the pressure sensor 22 may be positioned near the point 17 of the end cover 16. In some embodiments, the pressure sensor may be located in other parts of the body of the stylus and may be mechanically coupled to devices and/or members to effectuate the sensing of pressure.
In some embodiments, the pressure sensor may serve as a trigger for the capture, processing and/or transmission of location indicative images by the camera of the stylus. That is, the camera in the stylus 10 may take images of the pattern(s) on the touch screen in response to the pressure sensor registering an indication of pressure being applied to the tip of the stylus. In alternative embodiments, the tip may have a capacitive sensor therein and the stylus camera may capture images when the capacitive sensor detects a change in capacitance (presumably from coupling to a capacitive touch screen). In yet another embodiment, the computing device may be touch or pressure sensitive and may instruct the stylus to capture an image via a communication link between the device and stylus. Such instruction may be sent when the computing device registers a pressure or capacitive touch roughly corresponding to a pressure or capacitive profile of the stylus.
An antenna 24 is also provided to allow for the stylus to wirelessly transmit information to a host. The antenna 24 may be located any suitable location within the stylus 10 and, in some embodiments, may be located at an end opposite the point 17. Additionally, the stylus 10 may include an accelerometer(s) 36 and/or gyroscope 38 in some embodiments.
As illustrated, a microprocessor 32 may be coupled to each of the camera 18, the LED 20, the pressure sensor 22 and the antenna 24. In some embodiments, the microprocessor 32 may be configured to receive data from the components and transmit the raw data via the antenna 24 to a host. In other embodiments, the microprocessor 32 may be configured to filter and/or manipulate the data before transmitting the data to a host. For example, in some embodiments the microprocessor 32 may compute a directional vector and/or interpret patterns that have been perceived by the camera 18. Additionally, the microprocessor 32 may be coupled to a memory 34. The memory 34 may be configured to store operating instructions for the microprocessor 32. In some embodiments, the memory 34 may be configured to store data received from the components.
In some embodiments, one or more accelerometer(s) 36 may be provided within the stylus as well as a gyroscope 38. The accelerometer(s) 36 and gyroscope may be useful in determining the orientation of the stylus 10 and/or any movement of the stylus. The orientation and movement information provided by the accelerometer(s) 36 and the gyroscope 38 may be used for determining orientation and movement of the stylus relative to a host device.
The tablet computing device 40 may have a touch sensitive display 42 that provides a visual output to a user and may receive input from the user. Generally, the touch sensitive display 42 does not detect contact by the stylus 10. In particular, contact by the stylus 10 on the surface of the display 42 may not register as an input to the device 40. Rather, the stylus 10 may provide input to the device 40 via a wireless communication channel, such as an RF channel, an IR channel, or the like.
Patterns encoded on the surface of the display 42 or generated by the display 42 may be used to determine the location of any contact the stylus 10 makes with the display. In particular, the stylus 10 may capture images of the encoded patterns to either determine its relative location on the display 42 or may transmit image data to the device 40 so that the device may determine the location of the stylus contact.
A pattern, design and/or code may be provided that is not visible to a human eye, but may be captured by the camera of the stylus 10 to determine the relative location of the stylus 10 relative to the display 42. In some embodiments, the pattern may take the form of a QR code, a 2D code, a bar code and/or one or more other suitable encoding format.
In some embodiments, the pattern may be physically present in a layer positioned on or over the display. For example, the pattern may be created by lasers making dots or surface blemishes below a certain size that cannot be seen (e.g., approximately five microns in diameter). In some embodiments, an acid, chemical etch, or chemical deposition may be used to create the pattern.
In still other embodiments, the pattern may be created in a certain wavelength of light imperceptible to humans and the camera in the stylus 10 may be configured to capture light in a range that includes that wavelength. For example, the patterns may be printed with IR ink on glass. In some embodiments, the pattern may be created using a micropolarizer and the camera may have a polarizing filter to detect the pattern. For yet other embodiments, the polarization may include using patterned birefringent Indium-tin oxide (ITO) layer. In some embodiments, near field optics may be implemented to encode the display surface. In particular, small focal length optics near the pixels may be used.
In other embodiments, the pattern may be pixel based. That is, the pattern may be created by pixels illuminated in the display 42. In some embodiments, the pixels may generate the pattern in colors and/or at a rate that they do not impact a visual effect or graphic being shown on the display 42. In some embodiments, small variations in color may be provided using human eye color response mismatch to allow the changes to be imperceptible to the human eye. In some embodiments, small variations in pixel brightness may be used to encode location information. For example, in some embodiments, every third or fourth pixel may be change either its color, hue or the like imperceptibly to provide location information. That is, the pixel changes may be implemented to create a pattern perceivable by a camera but not to a human eye. In some embodiments, this may be due to the color change and/or the frequency at which the change takes place. In some embodiments, a gradient may be applied to the pixels to encode them with location information.
In some embodiments, the pixels may modulate to use the relatively slow temporal response of human eyes compared to the camera to hide the encoding from human observation. Fast modulation of the pixel between two states can appear to be the state in between. That is, a bright and dim modulation of a pixel make a medium pixel brightness, a red and blue modulation make a magenta pixel color, and so forth. The rapid changes can be captured by the camera, while the eye is incapable of resolving the changes and instead sees an average of the states. In some embodiments, the polarity of the light may be modulated to indicate location. That is, a layer that can dynamically change polarization at a pixel level can be used to employ a similar scheme. The camera can capture fast changes in the polarization, but the human eye does not see the changes.
It should be appreciated that one or more techniques may be implemented to encode location information for capture by the camera of the stylus 10. In some embodiments, light from the stylus 10 may be used to allow the pattern to be captured by the camera. Alternately or in addition, light from the display 42 may be used to enable and/or improve detection of the patterns by the camera. Each technique may be implemented in a manner that is imperceptible to a user. As one example, the light may be infrared light to illuminate an infrared pattern. As yet another example, the camera in the stylus may be particularly sensitive to changes in brightness and/or intensity of the pixels, to the extent that it may perceive changes that lie below the threshold of the human eye to perceive.
Additionally, the device 40 includes the 42. In some embodiments, the display 42 may take the form of a flat panel, graphic display, such as a liquid crystal display, or an organic light emitting diode display. In some embodiments, the display may have dedicated components 58 to help improve the rendering of graphics and/or to off load graphics processing from the processor 50. For example, a graphics processing unit 60 may be provided in some embodiments with a dedicated memory 62 for graphics rendering. Additionally, touch sensors 64 are provided so that the display is enabled as a touch screen display. In some embodiments, the touch sensor 64 may be capacitive touch sensors configured to sense contact by a conductive material, such as a finger.
A memory 66 is coupled to the processor 50. In some embodiments, the memory 66 may take the form of a random access memory (RAM), dynamic RAM, static RAM, or any other suitable type of memory including flash memory and read-only memory, for example. A storage memory 68 may also be coupled to the processor 50 and may be any suitable storage memory, such as a hard disk drive, semiconductor disk drive, tape drive, flash drive, etc. The storage memory 68 may store data, applications, programs, and/or an operating system for the device 40.
More particularly, the storage memory 68 may store programs related to the use of the optical stylus 10 as an input device for the device 40. For example, in some embodiments, the storage memory 68 may include stylus programs 70 related to the operation and interaction with the stylus 10 including routines for orientation 72, pattern generation 74, and position determination 76. In some embodiments, the orientation routine 72 may be directed to determining the orientation of the stylus 10 relative to the display 42. For example, data from the accelerometer of the stylus 10 may be compared with data from an accelerometer 53 of the device 40 to determine a relative angle of the stylus to the surface of the display.
The pattern generator routine 74 may be directed to generating a position indicating pattern with pixels shown on the display 42. The patterns generated by the generator 74 may be detected by the stylus 10 in order to determine the position of the stylus on the surface of the display.
The position determiner routine 76 may be used in conjunction with input from the stylus to determine the position of the stylus on the surface of the display 42. In some embodiments, the position determiner routine 76 may be configured to utilize patterns captured by the stylus to determine the position of the stylus.
As discussed above, the pattern 80 may be imperceptible to a user, but may be captured by the stylus 10. In some embodiments, the pattern may be encoded to provide relative positioning information. The pattern 80 is encoded with details as to its location on the display 42. That is, the pattern 80 may include x, y coordinates that may be read by the stylus. As long as the stylus is able to capture an image of the pattern 80, the location of the stylus relative to the surface of the display 42 may be determined. For example, the stylus may transmit the code or the location information to the device so that the device may know where the stylus is pointing relative to the surface of the display 42.
Additionally, in some embodiments, the pattern 80 may be encoded to provide orientation information so that the orientation of the stylus relative to the pattern may be discerned. In some embodiments, the pattern 80 may be rotation independent. In some embodiments, the pattern may be a barcode, such as a 2D barcode, a quick response (QR) barcode or may implement other encoding techniques.
Upon initiation of the pairing, the stylus 10 may obtain identifying (ID) information about the host (Block 104). For example, in some embodiments, the stylus 10 may request information from the host such as a media access control (MAC) address, or other unique identifier. In some embodiments, identifying information may be encoded in the display and, thus, may be detected by the stylus 10. The stylus 10 may also provide identifying information to the host (Block 106) and a communication channel may be opened between the stylus and host (Block 108).
It should be appreciated that the ID information may take different forms in different embodiments. Moreover, the ID information may be used in a way that allows for the communications between the paired devices to be recognized by each device. For example, in some embodiments, the host's ID information may be included in packets transmitted by the stylus 10. In some embodiments, the ID information may include encryption keys that may be exchanged and used to encrypt transmissions to help secure the communications between the paired devices. In some embodiments, the ID information may be obtained by capturing and image of the device or a portion of the host device that may be used to identify the host.
Upon opening the communication channel, the stylus may transmit data to the host (Block 110). In particular, the stylus 10 may transmit data collected by its camera, accelerometer, pressure sensor and so forth, to allow the host to determine the position of the stylus. In some embodiments, the stylus may be configured to transmit raw data, while in other embodiments, the stylus may transmit processed data (e.g., relative position data).
In some embodiments, the opening of the communication channel may trigger the display of encoded location indicia on the display of the host. Specifically, in some embodiments, the host may not continuously display encoded location indicia unless the host is paired for use with the stylus 10. Once paired with the stylus 10, the host may generate location indicia that may be captured by the camera of the stylus 10 but are imperceptible to a user.
Generally, as long as the stylus 10 remains in contact with the surface, image data will continuously be collected and transmitted to the host. It should be appreciated that the pressure sensor may be used in some embodiments to determine if the stylus is in contact with a surface. In some embodiments, however, other techniques may be used to determine if the stylus 10 is in contact with a surface. For example, resolution, size and/or ability to perceive the encoded data may be used to determine if the stylus 10 is in contact with the surface. In some embodiments, the camera of the stylus 10 may have a focal length that provides for clean capture of the encoded patterns when the stylus is in contact with the surface and otherwise may be unable to capture a meaningful image of the pattern.
In some embodiments, if it is determined that the stylus 10 is not in contact with the surface, movement and/or orientation data may still be obtained (Block 124). Additionally, it should be appreciated that, in some embodiments, movement and orientation information may be determined from the image data.
Moreover, because of the angle θ of the camera relative to the pattern, parallax may occur or the perception of the pattern may be skewed. In particular, portions of the pattern that are closer to the stylus may appear larger than those that are further from the stylus. As such, a square pattern may be captured as a trapezoid, for example. In some embodiments, pattern recognition software may be implemented in order to recognize the pattern or to correct any skewing of the pattern due to the angle at which the stylus is held relative to the display.
In some embodiments, information obtained from the accelerometer and/or gyroscope of the stylus may be used to determine the relative angle of the stylus and, thus, facilitate reconstruction of the captured pattern from a skewed pattern. Additionally, in some embodiments, the accelerometer and/or gyroscope information may be used to determine the relative orientation of the stylus to the display 42 so that obtained patterns may be reconstructed and/or interpreted properly. That is, for example, if a pattern is captured upside down (because of the orientation of the display or the stylus), the pattern may be read/interpreted right side up. Moreover, in some embodiments, the patterns themselves may include orientation information that may be used to correct for any orientation differential between the stylus and the display. For example, each code may include a particular feature (e.g., pattern or color change) that indicates an orientation of the pattern (such as a top of a pattern or a particular corner of the pattern).
The foregoing discussion describes some example embodiments of an optical stylus and a host computing system. Although the foregoing discussion has presented specific embodiments, persons skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the embodiments to achieve the similar security provided by the embodiments disclosed herein. Accordingly, the specific embodiments described herein should be understood as examples and not limiting the scope of the disclosure.
Number | Name | Date | Kind |
---|---|---|---|
4318096 | Thornburg et al. | Mar 1982 | A |
4695680 | Kable | Sep 1987 | A |
4761089 | Kurihara et al. | Aug 1988 | A |
4814552 | Stefik et al. | Mar 1989 | A |
4859080 | Titus et al. | Aug 1989 | A |
4883926 | Baldwin | Nov 1989 | A |
5247137 | Eppeerson | Sep 1993 | A |
5434371 | Brooks | Jul 1995 | A |
5581052 | Padula et al. | Dec 1996 | A |
5600348 | Bartholow et al. | Feb 1997 | A |
5736687 | Sellers | Apr 1998 | A |
5831601 | Vogeley et al. | Nov 1998 | A |
5914708 | LaGrange et al. | Jun 1999 | A |
6050735 | Hazzard | Apr 2000 | A |
6081261 | Wolff et al. | Jun 2000 | A |
6130666 | Persidsky | Oct 2000 | A |
6188392 | O'Connor et al. | Feb 2001 | B1 |
6211521 | Bawolek | Apr 2001 | B1 |
6650320 | Zimmerman | Nov 2003 | B1 |
6717073 | Xu et al. | Apr 2004 | B2 |
6914596 | Liu et al. | Jul 2005 | B2 |
6924791 | Nicolas | Aug 2005 | B1 |
6933933 | Fleming | Aug 2005 | B2 |
7027041 | Nishimura et al. | Apr 2006 | B2 |
7064750 | Duret | Jun 2006 | B2 |
7068262 | Perkins et al. | Jun 2006 | B2 |
7131724 | King et al. | Nov 2006 | B2 |
7310091 | Liu et al. | Dec 2007 | B2 |
7365735 | Reinhardt et al. | Apr 2008 | B2 |
7477242 | Cross et al. | Jan 2009 | B2 |
7489308 | Blake et al. | Feb 2009 | B2 |
7511705 | Silk et al. | Mar 2009 | B2 |
7612767 | Griffin et al. | Nov 2009 | B1 |
7646379 | Drennan et al. | Jan 2010 | B1 |
7671837 | Forsblad et al. | Mar 2010 | B2 |
7842893 | Tomiya | Nov 2010 | B2 |
7976226 | Jeon et al. | Jul 2011 | B2 |
8063322 | Katsurahira | Nov 2011 | B2 |
8089474 | Geaghan et al. | Jan 2012 | B2 |
8094325 | Silverbrook | Jan 2012 | B2 |
8130212 | Umeda | Mar 2012 | B2 |
8212795 | Henry et al. | Jul 2012 | B2 |
8259090 | Chiang et al. | Sep 2012 | B2 |
20020048404 | Fahraeus et al. | Apr 2002 | A1 |
20020067350 | Ben Ayed | Jun 2002 | A1 |
20020158854 | Ju | Oct 2002 | A1 |
20020181744 | Vablais et al. | Dec 2002 | A1 |
20030214490 | Cool | Nov 2003 | A1 |
20040140962 | Wang et al. | Jul 2004 | A1 |
20050099405 | Dietz | May 2005 | A1 |
20050110777 | Geaghan et al. | May 2005 | A1 |
20050110778 | Ben Ayed | May 2005 | A1 |
20050156915 | Fisher | Jul 2005 | A1 |
20050162411 | van Berkel | Jul 2005 | A1 |
20050206628 | Bich et al. | Sep 2005 | A1 |
20060087496 | Maciejewski | Apr 2006 | A1 |
20060123362 | Keely | Jun 2006 | A1 |
20060139338 | Robrecht | Jun 2006 | A1 |
20060140480 | Seto et al. | Jun 2006 | A1 |
20070076953 | Gonzalez et al. | Apr 2007 | A1 |
20070085842 | Pilu | Apr 2007 | A1 |
20070104372 | Wang et al. | May 2007 | A1 |
20070123165 | Sheynman et al. | May 2007 | A1 |
20070188477 | Rehm | Aug 2007 | A1 |
20070236474 | Ramstein | Oct 2007 | A1 |
20070285405 | Rehm | Dec 2007 | A1 |
20080036734 | Forsblad et al. | Feb 2008 | A1 |
20080036926 | Chen | Feb 2008 | A1 |
20080074401 | Chung et al. | Mar 2008 | A1 |
20080150917 | Libbey et al. | Jun 2008 | A1 |
20080150921 | Robertson et al. | Jun 2008 | A1 |
20080278443 | Schelling et al. | Nov 2008 | A1 |
20080309621 | Aggarwal et al. | Dec 2008 | A1 |
20090009489 | Lee | Jan 2009 | A1 |
20090032313 | Silverbrook et al. | Feb 2009 | A1 |
20090036176 | Ure | Feb 2009 | A1 |
20090115431 | Philipp | May 2009 | A1 |
20090167702 | Nurmi | Jul 2009 | A1 |
20090236153 | Kyung et al. | Sep 2009 | A1 |
20090251338 | Marggraff et al. | Oct 2009 | A1 |
20090273585 | De Haan | Nov 2009 | A1 |
20090314552 | Underwood et al. | Dec 2009 | A1 |
20100006350 | Elias | Jan 2010 | A1 |
20100051356 | Stern et al. | Mar 2010 | A1 |
20100170726 | Yeh et al. | Jul 2010 | A1 |
20100271312 | Alameh et al. | Oct 2010 | A1 |
20100315384 | Hargreaves et al. | Dec 2010 | A1 |
20110017838 | Delbaere | Jan 2011 | A1 |
20110162894 | Weber | Jul 2011 | A1 |
20110164000 | Pance | Jul 2011 | A1 |
20110221712 | Liang | Sep 2011 | A1 |
20110273376 | Dickinson et al. | Nov 2011 | A1 |
20110285670 | Li et al. | Nov 2011 | A1 |
20110291986 | Rebeschi et al. | Dec 2011 | A1 |
20120062497 | Rebeschi et al. | Mar 2012 | A1 |
20120098798 | Lee | Apr 2012 | A1 |
20120113065 | Chin | May 2012 | A1 |
20120228039 | Hinson et al. | Sep 2012 | A1 |
20130009907 | Rosenberg et al. | Jan 2013 | A1 |
20130050080 | Dahl et al. | Feb 2013 | A1 |
20130135220 | Alameh et al. | May 2013 | A1 |
20140078070 | Armstrong-Muntner | Mar 2014 | A1 |
20140078109 | Armstrong-Muntner | Mar 2014 | A1 |
Number | Date | Country |
---|---|---|
2445362 | Jul 2008 | GB |
20030035305 | May 2003 | KR |
Entry |
---|
“Birefringence in Liquid Crystals”, Sharp: Microelectronics of the Americas, http://www.sharpsma.com/lcds/lcd-reference-information/birefringence-in-liquid-crystals. |
Number | Date | Country | |
---|---|---|---|
20120127110 A1 | May 2012 | US |