Patent Application
20090231281
A computing system may provide a user with one or more mechanisms for receiving information from the computing system and one or more mechanisms for providing information to the computing system. As an example, information can be input to the computing system from a user with a mouse, track ball, writing tablet, keyboard, or other input mechanism. Furthermore, information can be output by the computer system to a user with a display screen, speakers, or other output mechanism.
The user experience provided by a computing system can be affected by the ease with which a user is able to provide the computing system with input and receive output from the computing system. In general, as the input and output processes become more transparent to the user, the user experience improves. In particular, well designed input and output systems allow new users to quickly master the input and output processes. However, in addition to being easy to learn, good input and output mechanisms do not handcuff advanced users from interacting with the computing system in a more advanced manner.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
The following Detailed Description describes a multi-touch virtual keyboard. The multi-touch virtual keyboard can be displayed by a computing system, thus providing information to a user. The multi-touch virtual keyboard is also used to facilitate touch input from a user, so that the user can provide the computing system with information. The multi-touch virtual keyboard includes two or more different keys, including at least one primary key and at least one modifier key. Each key of the multi-touch virtual keyboard is capable of receiving touch input by a user, and translating the touch input from the user into keyboard messages that can be used to pass information to various different aspects of a computing system. Touch input at only the primary key can be translated into a first keyboard message, and touch input at both the primary key and the modifier key can be translated into a second keyboard message, different than the first keyboard message.
The present disclosure is directed to virtual keyboards for use with multi-touch computing systems. As a non-limiting example, a virtual keyboard image can be displayed by a multi-touch computing system. The multi-touch computing system can process multi-touch inputs to the virtual keyboard image. The capacity to process multi-touch inputs may allow for a more natural and intuitive user experience as the operation of the virtual keyboard image may more closely resemble the operation of a standard, non-virtual keyboard.
Although shown in
As illustrated, a touch input 209 may be applied to a primary key 208 of virtual keyboard image 206. In response thereto, multi-touch computing system 100 may translate the touch input received at the primary key into a first keyboard message 210. A keyboard output 212 may then be displayed at display surface 102 as a text character within text box 106 of application window image 104.
Multi-touch computing system 100 may translate the temporally overlapping touch inputs received at the primary key and the modifier key into a second keyboard message 312, different than first keyboard message 210. A keyboard output 314 may then be displayed at display surface 102 as a text character within text box 106 of application window image 104.
The modifier key can modify the keyboard message of the primary key such that second keyboard message 312 is different than first keyboard message 210 and correspondingly, that keyboard output 314 is different than keyboard output 212. In other words, the combination of the primary key and the modifier key can be translated into a keyboard message and/or keyboard output that neither the primary key nor the modifier key generate independently. As used herein, the second keyboard message may be the combination of two or more individual keyboard messages. For example, touch input at primary key 208 may individually create a keyboard message “A,” and touch input at modifier key 310 may individually create a keyboard message “B.” In some embodiments, temporally overlapping touch input at primary key 208 and modifier key 310 may collectively create a keyboard message “A+B,” while in other embodiments, a keyboard message “C” may be created responsive to the temporally overlapping touch input. Both keyboard messages “A+B” and “C” are different than keyboard message “A” alone. As a Nonlimiting example, the first keyboard message (e.g., “A”) may correspond with a lower case letter, and the second keyboard message (e.g., “A+B” or “C”) may correspond with an upper case letter.
Although shown as a combination of a letter key representing the primary key and a shift key representing the modifier key, a combination of two, three, or virtually any suitable number of temporally overlapping multi-touch inputs may also be processed by multi-touch computing system 100 to generate different keyboard messages and/or keyboard outputs. Also, in other embodiments, virtual keys other than a letter key and the shift key may be designated as the primary key or the modifier key. Nonlimiting examples of primary keys include letter keys, number keys, alphanumeric keys, command keys, system keys, and the like. Nonlimiting examples of modifier keys include shift keys, option keys, control keys, alt keys, function keys, and the like. In some embodiments, a virtual key may serve as a primary key in one key combination and as a modifier key in another key combination. Furthermore, in some embodiments, two or more modifier keys may be used to modify a primary key, with each additional modifier key resulting in a different keyboard message. For example, touch input at a primary key, a first modifier key, and a second modifier key can be translated into a third keyboard message, different than the first keyboard message and the second keyboard message. It should be understood that virtually any temporal combination of different key combinations can be used to generate different keyboard messages.
As a nonlimiting example, keyboards operating in foreign language modes can use different combinations of modifier keys with a common primary key to generate distinct characters. As an example, an “F” key may generate a first Japanese language character, an “F+Ctrl” key combination may generate a second Japanese language character, while an “F+Alt+Ctrl” key combination may generate a third Japanese language character.
Prior virtual keyboard technologies have not allowed for multiple temporally overlapping touch inputs to be combined into a keyboard message that can be the basis for the generation of a keyboard output that neither the primary key nor the modifier key generate independently. Rather, to create such a keyboard output, prior virtual keyboard technologies typically require that a first touch input to a first key is applied and released and that a second touch input to a second key is subsequently applied. Thus, current virtual keyboard technologies are not capable of processing multiple touch inputs that overlap for a duration of time. Prior technologies may therefore result in a less intuitive user experience and hence, virtual keyboard use that is less time efficient.
Similarly, virtual keyboard image 418 is receiving a touch input 415 at primary key 420 and a touch input 417 at modifier key 422. Furthermore, touch input 415 at primary key 420 and touch input 417 at modifier key 422 temporally overlap. Multi-touch computing system 100 may translate the touch input received at the primary key and the modifier key into a keyboard message 424. A keyboard output 428 of keyboard message 424 may then be displayed at display surface 102 as a text character within text box 426 of application window image 416.
Multiple touch inputs at virtual keyboard image 406 and virtual keyboard image 418 can be independently translated by multi-touch computing system 100 into different keyboard messages, keyboard message 412 and keyboard message 424, and into corresponding keyboard outputs, keyboard output 413 and keyboard output 428. Additionally, the touch inputs at an individual virtual keyboard image temporally overlap with each other and may temporally overlap with the touch inputs at another virtual keyboard image. Furthermore, each keyboard message may be received by different temporally overlapping applications. In this manner, two or more users can use the same multi-touch computing system to effectively operate two or more applications at the same time, and each application can receive fully functional multi-touch keyboard input. Furthermore, two or more different users may use two or more different virtual keyboard images to control the same application in some embodiments.
As illustrated in
As an extension of the capacity to receive and process multiple touch inputs at a single virtual keyboard image, the capacity of multi-touch computing system 100 to receive and process multiple temporally overlapping touch inputs at more than one virtual keyboard image may allow for a more fluid and intuitive collaborative work experience for multiple system users. Efficiency of individual and collaborative work efforts may thus be improved.
At 504, touch input may be received by a primary key of the virtual keyboard image. At 506, it may be decided whether touch input is being received at a modifier key of the virtual keyboard image at the same time that touch input is being received by the primary key. If touch input at the modifier key is not being received at the same time that touch input is being received at the primary key, then a first keyboard message is created at 508. If touch input at the modifier key is being received at the same time that touch input is being received at the primary key, then a second keyboard message, different than the first keyboard message, is created at 510.
While the present disclosure uses a surface computing device as a non-limiting example of a multi-touch device capable of displaying a virtual keyboard, it should be understood that other multi-touch devices can be used in accordance with the present disclosure. It should be appreciated that the concepts disclosed herein may be implemented on any suitable touch-enabled display device that is capable of displaying a virtual keyboard and is also capable of processing two or more different user inputs having overlapping durations.
As used herein, the term “computing system” may include any system that electronically executes one or more programs. The embodiments described herein may be implemented on such a system, for example, via computer-executable instructions or code, such as system software or applications, stored on computer-readable media and executed by the computing system. Generally, such instructions include routines, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types.
The term “instructions” as used herein may connote a portion of a larger system or application, a single program, and/or multiple programs acting in concert, and may be used to denote applications, services, or any other type or class of logic executable by the computing system. Instructions can be implemented as software, firmware, or virtually any other form of executable logic. It should be appreciated that computer-readable media may include instructions which, upon execution by a processing subsystem, provide the virtual keyboard functionality described herein.
Display surface 602 may also be capable of receiving input from one or more users. For example, the multi-touch surface computing system can recognize the touch of a user, and can translate the various ways in which a user touches the display surface into different commands. Additionally, the multi-touch surface computing system can recognize the touch of a user by visually monitoring the display surface with one or more optical sensors, as described below in more detail. In other embodiments, the display surface may include sensors configured for capacitive touch sensing, resistive touch sensing, and/or another type of touch sensing.
As shown in
A portion of the instructions embodying the shell may optionally ensure that shell-level keyboard functionality is provided to only a single application at any given time (with regard to a single virtual keyboard image) and that a touch input received at display surface 602 (i.e. touch input at a text box within another open application window image) may allow keyboard functionality to be switched to another application.
Multi-touch surface computing system 700 may include a reference light source 726. A pattern of reflection of the reference light emitted by reference light source 726 may change responsive to touch input on display surface 706. For example, light emitted by reference light source 726 may be reflected by a finger or other object used to apply touch input to display surface 706. The use of infrared LEDs as opposed to visible LEDs may help to avoid washing out the appearance of projected images on display surface 706.
Reference light source 726 may be positioned at any suitable location within multi-touch surface computing system 700. As illustrated in the depicted embodiment, reference light source 726 may be configured as multiple LEDs that are placed along a side of display surface 706. In this location, light from the LEDs can travel through display surface 706 via internal reflection, while some light can escape from display surface 706 for reflection by an object on the display surface 706. In alternative embodiments, one or more LEDs may be placed beneath display surface 706 so as to pass emitted light through display surface 706.
Multi-touch surface computing system 700 may further include a sensor 724 that may be configured to sense objects providing touch input to display surface 706. Sensor 724 may be configured to capture an image of the entire backside of display surface 706. Additionally, to help ensure that only objects that are touching display surface 706 are detected by sensor 724, diffuser screen layer 714 may help to avoid the imaging of objects that are not in contact with or positioned within a few millimeters of display surface 706.
Sensor 724 can be configured to detect the pattern of reflection of reference light emitted from reference light source 726. The sensor may include any suitable image sensing mechanism. Examples of suitable image sensing mechanisms include, but are not limited to, CCD and CMOS image sensors. Further, the image sensing mechanisms may capture images of display surface 706 at a sufficient frequency to detect motion of an object across display surface 706.
Sensor 724 may be configured to detect multiple touch inputs. Sensor 724 may also be configured to detect reflected or emitted energy of any suitable wavelength, including but not limited to infrared and visible wavelengths. To assist in detecting touch input received by display surface 706, sensor 724 may further include an additional reference light source 726 (i.e. an emitter such as one or more light emitting diodes (LEDs)) positioned to direct reference infrared or visible light at display surface 706.
Multi-touch surface computing system 700 may further include processing subsystem 720. Processing subsystem 720 may be operatively connected to image generation subsystem 702 and sensor 724. Processing subsystem 720 may receive signal data from sensor 724 representative of the pattern of reflection of the reference light at display surface 706. Correspondingly, processing subsystem 720, may process signal data received from sensor 724 and send commands to image generation subsystem 702 in response to the signal data received from sensor 724. Furthermore, as illustrated by dashed-line connection 725 between display surface 706 and processing subsystem 720, display surface 706 may alternatively or further include an optional capacitive, resistive, or other electromagnetic touch-sensing mechanism.
Multi-touch surface computing system 700 may further include memory 718 that may be operatively connected to processing subsystem 720. Memory 718 may include a variety of different types of computer-readable media. Non-limiting examples of computer-readable media include one or more hard disks, one or more random access memory modules, one or more read-only memory modules, and removable media such as compact disks, digital versatile disks, Flash drives, and the like. Memory 718 may further include instructions. A portion of the instructions of memory 718, when executed by processing subsystem 720, may cause image generation subsystem 702 to project a virtual keyboard image at display surface 706. The virtual keyboard image projected by the image generation subsystem may include a primary key and a modifier key.
The instructions of memory 718 may further include a portion that, when executed by processing subsystem 720, may translate the pattern of reflection created responsive to touch input at only the primary key into a first keyboard message. Similarly, the instructions may further include a portion that, when executed by processing subsystem 720, may translate the pattern of reflection of the reference light created when temporally overlapping multi-touch input is applied at the primary key and the modifier key into a second keyboard message that is different than the first keyboard message. Also, the instructions of memory 718 may further include a portion that, when executed by processing subsystem 720, may provide shell-level virtual keyboard functionality to a plurality of different applications of multi-touch surface computing system 700.
Multi-touch surface computing system 800 may include a processing subsystem 820. Processing subsystem 820 may be operatively connected to image generation subsystem 802. Multi-touch surface computing system 800 may further include a reference light source 826. As illustrated, reference light source 826 may be configured as an LED array positioned to direct reference light (i.e., reference infrared or visible light) at display surface 806. Multi-touch surface computing system 800 may further include sensors 824a-824e. Sensors 824a-824e may be operatively connected to processing subsystem 820 and may be configured to detect the pattern of reflection of reference light at display surface 806.
Sensors 824a-824e may each be configured to capture an image of a portion of display surface 806 (i.e. detect multi-touch input to display surface 806) and provide the image to processing subsystem 820. Processing subsystem 820 may assemble a composite image of the entire display surface 806 from the individual images captured by sensors 824a-824e. Sensors 824a-824d may be positioned generally beneath the corners of display surface 806, while sensor 824e may be positioned in a location such that it does not pick up glare from reference light source 826 that may be reflected by display surface 806 and picked up by sensors 824a-824d. In this manner, images from sensors 824a-824e may be combined by processing subsystem 820 to produce a complete, glare-free image of the backside of display surface 806. Additionally, display surface 806 may alternatively or further include an optional capacitive, resistive or other electromagnetic touch-sensing mechanism, as illustrated by a dashed-line connection 825 of display surface 806 with processing subsystem 820.
Multi-touch surface computing system 800 may further include memory 818 that may be operatively connected to processing subsystem 820, image generation subsystem 802, and sensors 828a-828e. Memory 818 may include a variety of different types of computer-readable media. Non-limiting examples of computer-readable media include one or more hard disks, one or more random access memory modules, one or more read-only memory modules, and removable media such as compact disks, digital versatile disks, Flash drives, and the like. The computer-readable media of memory 818 may further include instructions. A portion of the instructions of memory 818, when executed by processing subsystem 820, may cause image generation subsystem 802 to project a virtual keyboard image at display surface 806. The virtual keyboard may include a primary key and a modifier key.
The instructions of memory 818 may further include a portion that, when executed by processing subsystem 820, may translate the pattern of reflection created responsive to touch input at only the primary key into a first keyboard message. Similarly, the instructions may further include a portion that, when executed by processing subsystem 820, translate the pattern of reflection of reference light created when temporally overlapping multi-touch input is received at the primary key and the modifier key into a second keyboard message that is different than the first keyboard message. Also, the instructions of memory 818 may further include a portion that, when executed by processing subsystem 820, provides shell-level virtual keyboard functionality to a plurality of different applications of multi-touch surface computing system 800.
It should be appreciated that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. For example, while described herein in the context of a multi-touch surface computing system having a horizontal, table-like display surface, it may be appreciated that the concepts described herein may also be used with display surfaces of any other suitable size and/or orientation, including vertically arranged display surfaces.
Furthermore, the specific routines or methods described herein may represent one or more of any number of processing strategies such as event-driven, interrupt-driven, multi-tasking, multi-threading, and the like. As such, various acts illustrated may be performed in the sequence illustrated, in parallel, or in some cases omitted. Likewise, the order of any of the above-described processes is not necessarily required to achieve the features and/or results of the exemplary embodiments described herein, but is provided for ease of illustration and description.
The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
