Patent Application
20080129694
Not applicable.
Not applicable.
This invention relates generally to the field of computer input devices, and more particularly to devices for transforming kinetic motions of the hands and wrists into symbols or control signals for use by a computer. Commonly recognized devices for performing these transformations include computer keyboards and computer mouse input devices.
A computer keyboard is a computer input device which provides a computer with a stream of discrete or superposed symbols. Depending on the computer program in operation to interpret the stream of symbols, these symbols may be treated as information to be stored for later retrieval or as commands to control the operation of the computer. Examples of the former treatment include typing a paragraph of English into a word-processing program; examples of the later treatment include typing the superposition of Control-Alt-Delete at the prompt of a computer running the MS-DOS operating system.
A computer mouse is used as a computer input device to control the location of a cursor on a video display connected to the computer. Information describing the movement of the mouse across a surface, or within a sensor volume, is provided to the computer where it is transformed into a corresponding cursor movement. In addition, there are typically two or three buttons on the mouse for providing discrete input.
Computer keyboards and mice provide a computer interface via motions of a user's hands and wrists, but a variety of alternative methods for providing computer input are known in the prior art which use, inter alia, motions of the mouth and tongue (“Mouth mounted input device” U.S. Pat. No. 7,071,844), the nose (“Method for video-based nose location tracking and hands-free computer input devices based thereon” U.S. Pat. No. 6,925,122), heart rate, temperature, general somatic activity and galvanic skin response (“Computer input device with biosensors for sensing user emotions” U.S. Pat. No. 6,190,314), the vocal tract (“Input device for computer speech recognition system” U.S. Pat. No. 4,461,024), the eye (“Eye tracking apparatus and method employing grayscale threshold values” U.S. Pat. No. 5,481,622), and so on.
Body-wearable devices, and glove-based devices in particular, for human-computer interaction are known to the prior art:
U.S. Pat. No. 3,022,878 to R. Seibel, of Feb. 27, 1962 discloses a data input device comprising a glove-like casing having a plurality of multi-position switches. The switches are set by the user's phalanges to various character representing positions in order to transmit data to an associated machine. This bulky device was designed for special purpose applications such as airplane cockpits and has proved to offer little of the functionality needed in more modern computer interfaces.
U.S. Pat. No. 4,414,537, filed Sep. 15, 1981, by G. Grimes and entitled “Digital Data Entry Glove Interface,” describes a glove-based input device. The Grimes patent discloses a glove with sensors for detecting the flexing of finger joints, sensors for detecting contact between various portions of the hand, and sensors for detecting the orientation of the hand. The Grimes device is used to identify static hand positions representing the characters of the alphabet. Although the Grimes device represents an advance over prior devices in terms of its reliability and streamlined form, it was designed particularly to process a single set of gestures called the Single Hand Manual Alphabet for the deaf and, given its limited programmability and set of fixed binary-response sensors, it is incapable of adaptation to different character sets or reprogramming to accommodate the desires of individual users. As such, it is not in general use today.
U.S. Pat. No. 4,988,981 for a computer data entry and manipulation apparatus and method by Thomas G. Zimmerman and Jaron Z. Lanier, patented Jan. 29, 1991, describes an “Apparatus . . . for generating control signals for the manipulation of virtual objects in a computer system according to the gestures and positions of an operator's hand or other body part.” This apparatus has a fixed vocabulary of gestures intended for use in controlling a cursor and manipulating virtual objects; as such it does not provide discrete alphanumeric input.
Although alternative methods of computer input such as described in the aforementioned U.S. Patents have found application in certain niches, the combination of computer keyboard and computer mouse remains the most common method used for human-computer interaction. However, this combination of two fundamentally different interface devices imposes an inefficiency on the user when the user must switch from one hand configuration to operate the traditional keyboard into another hand configuration to operate the mouse.
Portable computers are commonly sold equipped with a touch-pad (for a recent example of advances in this technology, see US Pat Appl #20060044259) and a traditional keyboard mounted together. The proximity of the two input devices alleviates the inefficiencies involved in switching between the interface devices to some extent, but some find these devices awkward to use and difficult to master.
Moreover, as discussed by Holzrichter et al. in U.S. Pat. Appl 20020033803 an important issue afflicting mouse-type user interface devices is that the design of these devices causes repetitive motion injury to many users. These injuries appear to occur because the mouse-motion on a plane and the location of the attached buttons is incompatible with natural hand-wrist-finger motions.
One solution to these problems is to integrate the functions of a computer mouse with the user's hand in a wearable glove. A recent attempt to do this is described in U.S. Pat. Nos. 5,444,462, and 6,097,369 issued to Wambach on Aug. 22, 1995 and Aug. 1, 2000, respectively. Wambach describes a glove to be worn on a user's hand wherein the glove includes micro-switches mounted next to a joint of the index finger and on opposite sides of the wrist.
Another recent and related invention is described in U.S. Pat. No. 6,154,199 issued to Butler on Nov. 28, 2000. Butler describes a hand positioned mouse which includes a glove having a trackball supported in a housing attached to the side of the index finger so that the trackball can be operated by the thumb.
Another recent glove-type user interface device is described in U.S. Pat. No. 7,057,604 issued to Bajramovic. Bajramovic describes a computer mouse on a wearable glove, which includes a tracking device for controlling cursor movement on a video display and one or more switches for controlling mouse “click” functions. The user of this device may type on a keyboard with all fingers while wearing the glove.
The inventions disclosed in U.S. Pat. Nos. 5,444,462, and 6,097,369 and 6,154,199 and 7,057,604 mitigate some of the ergonomic difficulties afflicting users of keyboard/mouse user interfaces; however, since they only address the design of the mouse interface without addressing the design of the keyboard interface, they do not represent a truly integrated keyboard/mouse solution. What is needed is a principled integration of keyboard and mouse functionalities within a low-cost, ergonomically sound design. The present invention provides a method and apparatus for achieving this integration. Moreover, the method and apparatus of the present invention adds a capability to human-computer interaction without precedent in the prior art: that of user-adaptability. In one of its embodiments the present invention is flexible enough to learn the preferences and habits of its users so that, over time, the performance of the interface will improve.
The present invention provides a glove-type computer user interface which performs all the functions of a computer keyboard and a computer mouse or touch-pad.
The invention can be characterized as an apparatus and set of methods, as implemented in a set of computer programs, for translating gestures made by a user's hands and wrists into computer-readable symbols and commands. The apparatus is comprised of a pair of light-weight gloves into which a plurality of sensors is embedded. Information gathered by the sensors representing the position, speed, and acceleration of the joints of the hand and wrist is transmitted to a computer by means of additional electronic apparatus. Using a gestural dynamical model, the computer implements methods to interpret such information in terms of gestures the user is trying to execute. These gestures are then mapped, in a user-definable fashion, onto some set of symbols and commands.
In an optional embodiment of this invention, the gestural dynamical model used to detect symbols and commands within the data stream can be made self-modifying so that the model evolves according to the user's dynamical habits. For instance, if a user habitually types a firm ‘y’ with the index finger of the right hand, but tends to type a gentle ‘j’ with the same finger, the dynamical model may use this firm/gentle distinction as a distinguishing characteristic between ‘y’ and ‘j’ for the user, even in the case that the user does not tend to make a spatial distinction between the two letters. Such a capability is not available in a traditional, fixed-position keyboard.
In yet another optional embodiment of this invention, the gestural dynamical model used to detect symbols and commands within the data stream can be made self-modifying so that the model uses historical and contextual information gathered by observation of a user's gestural habits to disambiguate otherwise ambiguous gestures. For instance, a user may tend to have a different hand position for an ‘h’ when preceded by a ‘t’ than when preceded by a ‘c’.
Some distinctions between this invention and a conventional keyboard are that:
The accompanying drawings, which are incorporated into and constitute a part of the specification, illustrate specific embodiments of the invention and, together with the general description of the invention given above, and the detailed description of some specific embodiments, serve to explain the principles of the invention by way of example without foreclosing such modifications and variations as would be apparent to a person skilled in the relevant arts.
In the following description, numerous specific details are set forth such as examples of specific components, processes, algorithms, etc. in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that these specific details need not be employed to practice the present invention. In other instances, well known components or methods have not been described in detail in order to avoid unnecessarily obscuring the present invention.
A collection of successive dynamic states measured while a user is performing an action intended to evoke a symbol or computer command is a gesture. The treatment of gestures as a collection of states or trajectory within a dynamical state space is termed gestural dynamics. For further discussion see Other References 2-6.
In a preferred embodiment the sensors 101 are flexible polymer piezoelectric accelerometers and strain gauges available from MSI Sensors of Hampton, Va.
In an alternative embodiment the wearable glove 100 is located within an ultrasonic or electromagnetic field (for example, the devices of U.S. Pat. Nos. 5,414,256 or 5,510,800 or 5,661,490). The plurality of sensors 101 is replaced by a plurality of passive or active means in a manner apparent to persons versed in the art which interact with said field to enable the measurement of the dynamical state of the wearable glove 100 from analysis of the ultrasonic or electromagnetic field.
In an alternative embodiment the wireless receiver 202 and wireless transmitter 203 are replaced by a direct physical wired connection from the wearable gloves 200L and 200R to the host computer 203.
As shown in the flowchart of
It is an advantage of the present invention that the gestural model 301 can be a standardized gestural model (such as might be obtained from the gestures involved in typing on a traditional qwerty keyboard) or it can be modified as needed by the user. In particular, although the illustration in
This flexible nature is further exploited in an alternative embodiment of the method of the present invention as illustrated in
Any number of embodiments of the feedback mechanism 404 may suggest themselves to persons skilled in the art. One such embodiment uses a technique suggested in Other References 7.
Said feedback mechanism 404 is an advantage of the present invention over current art in that a gestural model 401 adapted via said feedback mechanism enables the present invention to improve its functionality with time in terms of ease of use, speed of human-computer interaction, and ergonomic comfort.
In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
This application claims the benefit of co-pending U.S. Provisional Application Ser. No. 60/867,962 filed 30 Nov. 2006.
| Number | Date | Country | |
|---|---|---|---|
| 60867962 | Nov 2006 | US |
