Patent Application
20140168173
A stylus pointing device enables information to be input to an electronic device, such as a personal computer, tablet computer, smart-phone, or personal digital assistant. In particular, a stylus may be used in a conjunction with a display. Location coordinates provided by the stylus may be mapped to location coordinates on the display. In order that the location of the stylus corresponds accurately with the mapped location on the display, a calibration procedure may be performed.
In one calibration procedure, calibration images (such as dots) are rendered to indicate one or more locations on the display and a user is instructed to touch the stylus to the display at the indicated locations. However, this approach is inconvenient and the calibration accuracy is dependent upon how close the user moves the stylus to each indicated location.
Light pens use a mono-chromatic photo-detector to detect light displayed on the screen of a cathode ray tube (CRT). In a CRT, a single beam of electronics is moved across a phosphorescent screen in a raster scan. The estimated location of the beam at the time when light is detected by the pen is indicative of the location of the pen on the screen. This approach does not work with liquid crystal displays, however, since many pixels are illuminated at any given time, and any particular pixel may have constant illumination for extended durations. Even with raster scanned screens, the light pen may be located at a dark portion of the screen and be undetected.
In a second calibration procedure, a calibration image or pattern is periodically generated in the vicinity of the mapped location of the display. The calibration pattern may be a sequence of dots at increasing distances from the mapped location. When the image is detected by photo-sensor in the tip of the stylus, the actual position of the stylus is known. This approach is similar to the light pen approach, in that a time-varying illumination is used. However, the approach has the disadvantage of generating a calibration pattern that is visible to a user and may interfere with another image rendered on the screen.
Accordingly, it would be useful to provide a method and apparatus for stylus calibration that requires no user interaction, is not visible to the user, and may be used with pixelated display screens.
Some embodiments of the present disclosure will be described below with reference to the included drawings such that like reference numerals refer to like elements and in which:
For simplicity and clarity of illustration, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. Numerous details are set forth to provide an understanding of the illustrative embodiments described herein. The embodiments may be practiced without these details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the disclosed embodiments. The description is not to be considered as limited to the scope of the embodiments shown and described herein.
A stylus is designed to convey the location of the tip of the stylus on a surface, such as a tablet or screen, to a host electronic device. The host electronic device may be a portable computer, a handheld computer, a Personal Digital Assistant, a mobile cellular telephone, a smart-phone, a desktop computer, a tablet or convertible computer or the like. The location of the stylus may be determined by a variety of methods, including detecting the interaction of the stylus with the electrical properties of the screen (electromagnetic induction, electrical resistance, electrical capacitance), the optical properties of the screen, or by ultrasonic location, or by other sensing means. These methods use sensors or transducers to convert physical phenomena into electrical signals. The electrical signals are then used to estimate the stylus location. The estimation utilizes knowledge of calibration parameters that may be dependent upon the sensitivity and/or locations of the sensors, for example. If the calibration parameters are inaccurate, the estimate of the stylus location will also be inaccurate. Calibration parameters may change over time, so it is desirable that calibration parameters be re-measured periodically. Examples of calibration parameters include detector sensitivities and offset, physical properties such as material properties, and properties that depend on environmental parameters. For example, an ultrasonic detection system uses knowledge of the propagation speed of ultrasound in the vicinity of the electronic device. This varies with environmental properties such a temperature and humidity, for example.
In accordance with certain aspects of the present disclosure, calibration parameters are automatically re-measured during normal interaction between a stylus and a graphical user interface displayed on a screen of a host device, without the need for a dedicated calibration process.
In accordance with exemplary embodiments, a color image, comprising an array of pixels, is displayed on the screen. A first screen location is determined at which one or more color components of a pixel of the displayed color image match color components sensed by the stylus. A second screen location is determined in response to a sensed location of a stylus and dependent upon one or more calibration parameters. If the first and second locations are different, the calibration parameters are adjusted dependent upon the first and second screen locations.
In general, a color image comprises a mapping from the screen locations of pixels to the color components of the pixels. In accordance with certain aspects of the disclosure, the color image is selected such that there is a mapping from color components to screen locations. For most images, such a mapping does not exist, since the same color may be displayed at multiple locations. However, images may be designed for which the mapping does exist.
A stylus is commonly used in conjunction with a graphical user interface to enable user input to the host electronic device. The graphical interface may include one more regions in which a graphical control, such as an icon, button or tile, is displayed. The host device is controlled, at least in part, by user interaction with graphical controls.
Certain aspects of the disclosure relates to an electronic device that includes a screen operable to display a color image formed by a plurality of pixels, each having one or more color components. The color image is defined by a mapping between screen locations and pixel color components. For certain images, there is a corresponding mapping between pixel color components and screen location. If this mapping exists, it enables a processor of the electronic device coupled to the screen to determine a first screen location at which the one or more color components of a pixel displayed at the first location matches a color value sensed by a stylus. In accordance with certain aspects of the present disclosure, the color image, or at least part of the color image, is selected such that a mapping between pixel color components and screen location exists. The processor may also determine a second screen location in response to a sensed location of the stylus and one or more calibration parameters. The one or more calibration parameters may then be adjusted in response to the first and second screen locations.
An image 106 comprises an array of color pixels. If each pixel or group of pixels in the array has a unique color, a measurement of the color may be used to determine a location within the array and a corresponding location on the display screen. In accordance with various aspects of the present invention, the stylus 102 incorporates a color sensor that is operable to sensor color of light emitted from the screen 108 in the region of the stylus tip 112.
The sensed color components may be communicated from the stylus 102 to the host electronic device 110 over a wired link or over a wireless communication link, such as a Radio-Frequency (RF) or Infrared (IR) link.
In accordance with certain aspects of the disclosure, a known mapping between the color components of a pixel, or group of pixels, of the image 106 and the screen location at which it is displayed is used to determine a stylus location with respect to the screen.
The image 106 may also contain text 208 or other graphics. In one illustrative embodiment, the text color (or colors) is not used in the background of the image. When the text color is sensed, no calibration is performed. Since much of the image is color coded for location, calibration is performed often. In a further embodiment, the text is also color coded for location, so that calibration may be performed whenever the stylus senses a color within the image. In yet another embodiment, a region of the image 106, such as the text region or another region, may be used to calibrate the color sensor of the stylus. For example, a white region may be used to set the white balance of the stylus sensor.
In operation, a color sensor in a stylus senses the color of a pixel or group of pixels (e.g., 210), and determines the color components. The color components may then be used to determine the location of the pixel having those color components using the known mapping between color components and screen location.
In a further operation, a second image such as a second icon, is provided with a white background to enable calibration of the color balance of the stylus.
C1 and C2 may be components of a two-dimensional color space, such as an r-g chromaticity color space. In such a color space, colors are normalized by an intensity level so that color measurements sensed by the stylus are independent of screen brightness. For example, if R, G and B are the red, green and blue color components of a pixel, the r-g chromaticity components are
In a further example, the color components are
Other two-dimensional color spaces may be used.
In the example shown, the x-coordinate of a position may be obtained by sensing the (normalized) amount of first color component and the y-coordinate may be found by sensing the (normalized) amount of the second color component. More generally, a lookup table indexed by color components may be accessed to find location coordinates. It will be apparent to those of ordinary skill in the art that other color components may be used.
In a further embodiment, a lookup table indexed by location coordinates is searched to discover the location of a measured color.
In principle, a one dimensional scale such as gray scale or intensity scale, can be used. For example, each pixel could have a unique gray value, allowing a mapping between gray-value and position, sensing the value would be difficult. Black and white regions may be provided to allow automatic calibration of a photo-sensor in the sensor with respect to the screen brightness.
The stylus 102 includes a color sensor 410 this is operable to detector color components of light at the tip of the stylus. The color components are passed to a communication circuit 412 and are transmitted via communication link 414 to a corresponding communication circuit 416 of the host electronic device 110.
The location of the stylus may be detected by various means known to those of ordinary skill in the art. In accordance with an exemplary embodiment, screen 108 is a touch screen coupled to a touch processor 418 that senses a touch location and passes a location signal 420 to the processor 402. In a further exemplary embodiment, the screen location is sensed by the stylus and communicated via the communication link 414 to the host electronic device 110.
In certain illustrative embodiments, an optical channel 502 is used to couple light from the tip 112 of the stylus 102 to the color sensor 410. The channel may be an open channel, or an optically transparent material and may be surrounded by a reflective coating.
An optical lens may be positioned in front of the color sensor to provide improved resolution.
In order to compensate for variations in screen brightness, the stylus may normalize one or more of the red, green and blue signals to be independent of brightness of the light at the stylus tip.
In certain illustrative embodiments, the first screen location is determined by accessing a lookup table indexed by the one or more color components. The one or more color components may be normalized to be independent of screen brightness.
For improved accuracy, the calibration parameters may be adjusted dependent upon color components sensed when the stylus is in contact with the screen. This may be achieved by sensing when the stylus is in contact with the screen and only calibrating the stylus when stylus is in contact with the screen.
For improved accuracy, a second color image having a uniform color background may be displayed on the screen. When the stylus senses the color components of the uniform background, the relative levels of the components may be compared to the known relative levels of the uniform background. This enables the color balance of the one or more color components sensed by the stylus to be adjusted dependent upon sensed color components of the uniform color background of the second color image.
Additionally an image have a spatially uniform color may be displayed. The known color components of the image may be used to calibrate to color sensor of the stylus. For example, a white image may be displayed, for which the known color components are equal.
The implementations of the present disclosure described above are intended to be merely exemplary. It will be appreciated by those of skill in the art that alterations, modifications and variations to the illustrative embodiments disclosed herein may be made without departing from the scope of the present disclosure. Moreover, selected features from one or more of the above-described exemplary embodiments may be combined to create alternative embodiments not explicitly shown and described herein.
It will be appreciated that any module or component disclosed herein that executes instructions may include or otherwise have access to non-transient and tangible computer readable media such as storage media, computer storage media, or data storage devices (removable or non-removable) such as, for example, magnetic disks, optical disks, or tape data storage. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the server, any component of or related to the network, backend, etc., or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media.
The present disclosure may be embodied in other specific forms without departing from its spirit or essential characteristics. The described exemplary embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the disclosure is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.
