OHAI technology user interface

[0002] The workstation user environment is characterized by sufficient surface room to put a computer, a mouse and other paraphernalia necessary to do a job of work involving a PC as a tool. In the workstation user environment, most user applications that require input assume a mouse and keyboard. Consequently, the process of input involves two hands manipulating these two input devices. To send input to applications, the user types with both hands, removes one hand from the keyboard, manipulates the mouse, and then returns the hand to the keyboard. While they are on the keyboard, the hands and fingers constantly move from key to key while the (typical) user attempts to watch the user application screens or windows and the keyboard simultaneously. One Handed Input, OHAI, technology is an alternative which allows a user to keep his attention on the screen by removing the possibility of erroneous input through misstruck keys.

SUMMARY OF THE INVENTION

[0003] The present invention relates to chord-based or syllable-based input systems and methods for presenting and delivering user application input choices and subsystems for generating application inputs in response to user generated combinations of finger-presses, hereinafter chords, or spoken syllables. OHAI denotes One Handed Input.

[0004] The present invention includes: systems, methods, and apparatus for presenting user application input choices, chord-based or syllable-based input apparatus, and sub-systems for generating application inputs in response to user chords or spoken syllables.

[0005] The new system for presenting and delivering user application input choices includes memory associating chord-sequences or syllable-sequences with complex user application input choices, an optional display associating chords or syllables with complex user application input choices, an input device for inputting chords or syllables, a first processor for generating input signals from chords or syllables, a second processor, connected to the first processor, for receiving sequences of input signals from the first processor, for evaluating the input signal sequences, connected to the optional display for modifying the display according to the evaluated input signal sequences, and connected the user application for sending user application input choices to the user application.

[0006] There are numerous potential embodiments of the input device. In one preferred embodiment, the input device resembles a joystick and includes a base and a key carrier extending from the base. The key carrier includes a bottom that is connected to the base, a top surface, a pair of side surfaces, a front surface and a back surface. Keys are provided along one or more of the surfaces. In preferred embodiments, a set of four keys extend widthwise along either a side surface or the front surface. A single key is provided along the top surface. The keys are positioned such that when a user grasps the device, the user's thumb rests on the button along the top and the user's fingers rest on the set of four keys. Each key may be depressed using a single finger or thumb without removing or repositioning the digits.

[0007] In another preferred embodiment, the input device is integral with a laptop computer running the user application. At least one set of depressible buttons is provided on the computer housing. In a related preferred embodiment, two sets of five ergonomically placed buttons are provided, with one set positioned on the keyboard section of the housing and another set provided on the monitor section of the housing.

[0008] When the user application resides on a wearable computer, one embodiment of the input device is a glove including sensors integrated into the fingertips of the glove. An alternative embodiment of the input device is a voice-input processor of sufficient sensitivity to distinguish among a small set of spoken syllables. The preferred embodiment of the display on wearable computers is a grid showing associations of chords or syllables which is displayed on a headset having a small monitor.

[0009] The first processor monitors user-generated chords or syllables and sends a corresponding input signal to the second processor. The first processor may include a chip housing a program and may be mounted in the input device.

[0010] The present method for presenting and delivering user application input choices includes the steps of relating complex user application input choices to sequences of chords or syllables in memory, optionally displaying associations of chords or syllables within sequences of chords or syllables associated with complex user application input choices, receiving user generated chords or syllables using an input device, generating input signals from the input chords or syllables, evaluating input signal sequences, modifying the optional display to prompt for chords or syllables within chord or syllable sequences associated with complex user application input choices, and sending complex user application input choices to user applications.

[0011] In preferred embodiments, the display is a grid arrangement of blocks including a grid name block and multiple chord blocks. Each chord block includes a row of finger cells and a large selection cell positioned near the row of finger cells. The finger cells of each block are coded such that a user may identify which combination of switches must be pressed and released to select the desired chord. Tentative sub-input choices may be made by pressing but not releasing, a combination of switches. If the chosen selection cell includes the name of a different grid, that successor grid replaces the current grid.

[0012] The display may be displayed on the computer monitor.

[0013] The present invention may be used for inputting Chinese or other foreign-based characters to computer user applications. For inputting Chinese characters, the method of character selection includes pinyin selection and hanzi selection. Pinyin selection includes initially selecting an appropriate pinyin initial, selecting an appropriate vowel cluster, and selecting an appropriate final/tone combination. Once the pinyin is selected, hanzi is displayed and selected.

[0014] Similar methods may be used for other languages.

[0015] The present invention can be used for word processing applications, for worldwide web and virtual reality navigation applications, for structured application development and operating system interface applications, and for menu navigation in applications that use menus.

[0016] These and further and other objects and features of the invention are apparent in the disclosure, which includes the above and ongoing written specification, with the claims and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIGS. 1-3 show side views of a (WOH) Workstation OHAI Handset.

[0018]
FIG. 4 shows a WOH grasped by a hand.

[0019]
FIG. 5 shows an OHAI grid for English text input

[0020]
FIG. 6 explodes a chord block of an OHAI grid and shows the relation between finger cells and the workstation OHAI handset.

[0021]
FIG. 7 is a workstation showing an OHAI window and a handset.

[0022]
FIG. 8 shows a WWW application window.

[0023]
FIG. 9 shows a WWW application window with OHAI Chordmap overlay.

[0024]
FIG. 10 illustrates an OHAI grid for OTIS (OHAI Text Input System).

[0025]
FIG. 11 shows the OTIS grid task specification.

[0026]
FIG. 12 shows the OTIS successor grids.

[0027]
FIG. 13 shows OTIS finite state automaton.

[0028]
FIG. 14 illustrates using OTIS to input English text.

[0029]
FIG. 15 shows OHAI system information flow.

[0030]
FIG. 16 is a chart of Mandarin Syllables.

[0031]
FIG. 17 is a table of Pinyin counts.

[0032]
FIG. 18 shows forty two hanzi with the same pinyin as “shi”.

[0033]
FIG. 19 shows the first TwinBridge prompt window for □.

[0034]
FIG. 20 shows TwinBridge prompt windows for □.

[0035]
FIG. 21 shows CHIME combined Pinyin initials and vowels.

[0036]
FIG. 22 is an overview of CHIME.

[0037]
FIG. 23 is a CHIME Pinyin chord sequence chart.

[0038]
FIG. 24 shows the four CHIME chords to produce □.

[0039]
FIG. 25 shows the five CHIME chords for □.

[0040]
FIG. 26 shows the two CHIME chords for □.

[0041]
FIG. 27 show examples of two-chord hanzi.

[0042]
FIG. 28 shows the three CHIME chords for □.

[0043]
FIG. 29 shows examples of three-chord hanzi.

[0044]
FIG. 30 shows the “Any Tone” grids for “shi”.

[0045]
FIG. 31 illustrates the integration of CHIME and OTIS.

[0046]
FIG. 32 shows an OTIS data file diagram.

[0047]
FIG. 33 is an OHAI firmware flowchart.

[0048]
FIG. 34 is an OHAI Gridset application flowchart.

[0049]
FIG. 35 is a front view of a laptop with OHAI capability.

[0050]
FIG. 36 is a back view of the OHAI laptop of FIG. 35.

[0051]
FIG. 37 shows an OHAI Glove for a left hand.

[0052]
FIG. 38 illustrates a headset for use with wearable computer.

[0053]
FIG. 39 shows a joystick-type OHAI handset.

[0054]
FIG. 40 shows chords and corresponding letters for non-display applications.

[0055]
FIG. 41 is a schema for a secure silent communications OHAI application.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0056] WOH: THE WORKSTATION OHAI HANDSET

[0057] OHAI denotes one-handed input. The Workstation OHAI Handset 1 (WOH) as shown in FIG. 1 either replaces or coexists with the keyboard in the workstation user environment. FIG. 1 shows a preferred embodiment in which the handset 1 resembles a joystick and includes a base 2 and a key carrier 3 extending from the base. The key carrier includes a top surface having a single key 4 and a set of four keys 5, 6, 7, 8 extending widthwise along a side surface. The keys are positioned such that when a user grasps the device as shown in FIG. 4, the user's thumb 10 rests on the top key 4 and the user's fingers rest on the set of four keys 5, 6, 7, 8. Each key may be depressed using a single finger or thumb without removing or repositioning the digits.

[0058] This one-handed device serves the same function as the keyboard. The WOH in FIG. 1 is for use with the left hand. To use it, the user grasps it so that the little finger is on the lowest button, each successive finger is placed on the next highest button and the thumb is placed on top. To send input to user applications, he pushes and releases combinations of buttons. Typically, a user keeps one hand on the WOH and one hand on the mouse throughout his session. Although it is recommended that the WOH be the only text input device, and the keyboard discarded, it is possible to use all three devices simultaneously. Some examples where this might be desirable are in the early stages of transition from keyboard to WOH, or if two extremely different input sets are used at the same time. For example, the WOH can be set to send Chinese input while the keyboard is being used for English input.

[0059] With the WOH, the user never has to take his hand off the device, never has to take his fingers off the keys, and never has to take his eyes off the screen. The WOH guides the fingers to their proper position and the pressable surfaces extend the full length of a face of the device, so no time or effort is wasted positioning the fingers. Since it has only five keys, the fingers don't move from key to key, so it is never necessary to remove the eyes from the user application. The design of the WOH alleviates many of the factors contributing to repetitive motion disorders such as carpal tunnel syndrome. The dimensions can be easily and inexpensively customized to individual users.

[0060] Additionally, a one-handed device frees up the other hand for other uses: mouse manipulation, telephoning, or holding the morning coffee cup.

[0061] The WOH is ideally suited to the workstation user environment either as an enhancement to, or as a replacement for, the keyboard. As an enhancement, it takes up very little room (less than a mouse, and considerably less than a mouse pad). The shape of the WOH allows it to be placed out of the way easily when not in use. The device is lightweight (approximately same as a mouse), so it can be picked up and used as well as being used with the forearm resting on the working surface. Under either circumstance, it can be positioned and repositioned to relieve stresses that contribute to repetitive motion disorders. In addition to benefits as an enhancement, as a keyboard replacement it frees even more surface area and provides more freedom of movement for the user.

[0062] The advantages of WOH in the workstation user environment are simplicity, comfort, and freedom to reposition in three dimensions as well as rotation. In short, the user can position it in any manner to suit. Moreover, it takes little physical exertion to use and does not require looking at to correctly position fingers on the device. Other forms of the device are equally well suited to other computing environments and will be disclosed hereafter.

[0063] HOW TO USE THE HANDSET

[0064] As shown in FIG. 4, the user holds the WOH so that the thumb rests on the top key, and the fingers rest on the four remaining keys. To select an input, the user presses and releases a combination of keys. Each combination is called a chord and the act of pressing and releasing a combination is called chording. There are only 31 chords, and mastering them with an OHAI training game takes only 1-3 hours (in contrast to the 100+ hours it takes to achieve partial mastery of a QWERTY keyboard).

[0065] OHAI NOTATION and VOCABULARY

[0066] Each chord can be represented notationally by a series of circles with darkened, numbered circles indicating pressed fingers and clear unnumbered circles indicating unpressed fingers. Since the example handset is left-handed, leaving the right hand free for precision mouse work, the fingers are numbered so that:

[0067] 1=Pressed Little Finger

[0068] 2=Pressed Ring Finger

[0069] 3=Pressed Middle Finger

[0070] 4=Pressed Index Finger

[0071] 5=Pressed Thumb

[0072] The chord consisting of all fingers (the full chord) is represented 12345, a chord consisting of just the thumb is 00005, a chord consisting of the little finger, index finger and thumb is 10045. Other chords are notated in like manner.

[0073] Each chord represents either an input choice or one in a sequence of sub-inputs, which taken together specify an input (like pressing the SHIFT or CAPLOCK key on the keyboard before pressing the letter to be capitalized). An OHAI window (as shown in FIG. 5) coexists with user applications windows on the workstation screen (see FIG. 7) to give the user chording instructions.

[0074] To show the user how to use the WOH, the OHAI window displays one of a number of OHAI grids which associate chords with user application inputs (as shown in FIG. 5). FIG. 6 breaks out and labels the parts of an OHAI grid for alphanumeric input.

[0075] As shown in FIG. 5, the grid 20 is a 4×8 arrangement of thirty two blocks 21. The upper left block 22 is the name block and gives the grid name which provides an overview of the choices presented by the grid. The remaining thirty one chord blocks 21 have two parts: a row 23 of rectangles on the top called the chord map containing five finger cells 31, 32, 33, 34, 35 and a larger rectangle 24 below called the selection cell.

[0076] In each chord block 21, the finger cells 31, 32, 33, 34, 35 are either dark or clear. A dark finger cell indicates a finger to be pressed and released. A clear finger cell indicates an unpressed finger. For example, a chord map consisting of all five darkened finger cells () represents the full chord (12345), a chord map with the first, fourth and fifth finger cells darkened () indicates the chord 10045, and in the grid 20 shown in FIG. 5 designates the lower case letter “r”. Other chords are similarly indicated by chord maps.

[0077] The range of input and sub-input choices immediately available is given by the grid currently displayed in the OHAI window. FIG. 7 shows for example, an a-z grid 26 displayed on a computer monitor screen 27. The selection cell indicates to the user what will happen when the chord in the chord map above it is chorded (pressed and released). Tentative (pressed, but not released) input choices are indicated by highlighted selection cells corresponding to unreleased chords.

[0078] OHAI APPLICATION TYPES

[0079] One of many disadvantages with previous implementations of chord-based input systems is that they only allow English text input. In contrast, OHAI applications associate chord-sequences with any user input set, or multiples thereof. This is especially useful in those tasks for which the keyboard is inappropriate to the input set, such as Chinese and Japanese text processing and worldwide web (WWW) and virtual reality (VR) navigation. In addition, OHAI carries with it the possibility of a unified user interface which provides a single mechanism to combine the functionalities of toolbars, pull-down menus, accelerator keys, and dialog boxes. These concepts fall into three intuitive categories of user application: word processing in any language is an example of an input-intensive user application; WWW Navigation is an example of content/choice user application; structured application development and operating system interfaces are examples of option-intensive user applications. Formal definition of these three categories is outside the scope of this disclosure; in general a word processor is distinct from a web-browser is distinct from a software development suite with respect to the choices their users make when using them. Each of these user application categories has a corresponding type of OHAI application.

[0080] OHAI applications fall into three general types: OHAI gridset applications, OHAI chordmap arrangements, and OHAI-based interfaces. OHAI gridset applications are used for input intensive user applications such as word-processors and spreadsheets. With OHAI Gridset Applications, the workstation screen has both an OHAI window and a user application window (FIG. 7). The user application window is maintained by the user application. The OHAI window is maintained by the OHAI processor software and displays one of a number of the grids which comprise the OHAI gridset application. OHAI chordmap arrangements are used for choice intensive applications such as WWW and VR navigation in which the user application presents options like WWW links or VR gestures. With OHAI chordmap arrangements, the user application window (FIG. 8) is overlaid with chordmaps (FIG. 9) which provide chord shortcuts to the choices presented therein. OHAI-based interfaces are complete user applications whose method for interacting with users are OHAI gridsets and chordmaps rather than the variety of user interface elements in Windows-Icons-Menus-Popups (WIMPS) applications; an example is ODS, which enables users to create and modify OHAI applications. Some examples of OHAI applications, and their types, are:

[0081] OTIS (OHAI Gridset Application, OHAI Text Input System)

[0082] CHIME (OHAI Gridset Application, CHinese Input Method)

[0083] HAJIME (OHAI Gridset Application, HAnd-based Japanese Input MEthod)

[0084] OWN (OHAI Chordmap Arrangement, OHAI Web Navigation)

[0085] OVR (OHAI Chordmap Arrangement, OHAI Virtual Reality)

[0086] OMI (OHAI Gridset Application, OHAI Mouse Interface)

[0087] ODS (OHAI Based Interface, OHAI Development System).

[0088] In OHAI gridset applications, such as OTIS and CHIME, the OHAI window displays grids whose form remains constant; there is always a name block and thirty one chord blocks each showing a chord map and a selection cell. Only the contents of selection cells change. For OTIS, the selection cells contain English text inputs and sub-inputs; for CHIME, they contain Chinese text inputs and sub-inputs. The sub-inputs are indicators of successor grids showing sequential input subsets. If the selection cell contains an input, such as a text character, that input is sent to the user application when the chord in its corresponding chord map is chorded. If the selection cell contains the name of a different grid, that successor grid replaces the current (predecessor) grid in the OHAI window, giving the user a further set of selection cell choices. Some selection cells both cause input to be sent to applications and have successor grids.

[0089] These concepts are illustrated by OTIS—a sample OHAI Text Input System.

[0090] OTIS: AN EXAMPLE OHAI APPLICATION

[0091] Introduction to OTIS

[0092] Almost every user application accepts and processes English text input. This means that an input system must have the ability to send the applications lowercase letters, uppercase letters, Arabic numbers, punctuation, and common symbols. OTIS is a sample (and modifiable) OHAI gridset application which accomplishes these input goals. This section presents and describes the components of OTIS, shows how to use OTIS, and then presents some design considerations and advantages, as well as some alternative layouts.

[0093] OTIS Description

[0094] The OTIS application comprises grids (FIG. 10), each of which accomplishes specific tasks. FIG. 11 defines the primary and secondary input tasks associated with each grid. The first, or base, grid for OTIS is the LOWER grid. The LOWER grid shows the user which chords input lowercase letters, spaces, and backspaces, as well as which chords select other grids. For example, the 00005 chord inputs a space, 02340 inputs lowercase ‘m’, 12345 inputs backspace, and 12300 selects the UPPER grid for capitalization. The chords for lowercase letters, space, and backspace have no successor grid; when these are chorded, the LOWER grid remains in the OHAI window. The chords for UPPER, NUMBER, and PUNCT all have successor grids named by their selection cells. These allow the user to choose from input sets of uppercase, numeric, or symbol characters. The ability to quickly choose a capital letter demonstrates the utility of successor screens. If the user wants to input a capital ‘O’, he will chord 12300 to bring up the UPPER grid, and then chord 10000 for ‘O’, after which its successor grid (LOWER) win replace the UPPER grid. OTIS behaves similarly for input of single numerals (the NUMBER chord block and grid) and symbols (the PUNCT chord block and grid).

[0095]
FIG. 12 explicitly shows the relationships between selection cells and their successor grids. The arrows in FIG. 12 show successor grids for each chord. Since arrows from each chord block would be confusing, the figure uses arrows from the side of the window to indicate the successor grid for all chords in a grid not explicitly indicated. For example, the arrows from the LOWER grid show that most of the chords have the LOWER grid itself as a successor grid, and the arrows from the UPPER grid show that all chords have different successor grids; in most cases LOWER. These are examples of selections that both cause input and have a successor grid. The LOWER and UPPER selection cells themselves being examples of selections which have successor grids (which they name) but cause no input.

[0096] Because FIG. 12 can be confusing, FIG. 13 is another representation of the same information. Each circle corresponds to an OTIS grid, the labels on each arc indicate inputs which OTIS will send to user applications when they are chorded, each arc goes to the successor grids for these inputs.

[0097] Using OTIS

[0098]
FIG. 14 shows how to chord the phrase “the OHAI handset”. Column 1 shows the contents of the OHAI window, column 2 shows chords, column 3 shows selections for the chords, column 4 shows the contents of a user application window after a chording sequence, column 5 shows keyboard keys corresponding to the chords. To send the text to a user application, chord 10305, 02005, 00340 and 00005 to input characters ‘t’, ‘h’, ‘e’, and a space. Chord 12300 twice to select the UPPER grid followed by the CAPLOCK grid. Chord 10000, 02005, 00040, 02040 for “OHAI”. Chord 12300 a third time to remove CAPLOCK and return to the base grid. Chord 00005 to input the space after “OHAI”. Finally, chord 02005, 00040, 02345, 00305, 10300, 00340, 10305 for “handset”.

[0099] OTIS Design Considerations & Advantages

[0100] Some of the issues to consider in designing OHAI gridset applications are: sequence minimization, intuitive chord-input association, input locatability, related function chord mapping, and chord difficulty.

[0101] Sequence minimization means that each input in a gridset should be accessible from the base grid (and as many other grids as possible) using the fewest sequential chords.

[0102]
FIG. 12 shows that OTIS minimizes sequence size by placing the names of other grids in the lower left three selection cells. This makes all grids accessible to all the others by at most two chords, so that all inputs result from at most two chords from the base. Moreover by putting the most common inputs (lowercase letters) on the OTIS base grid, the most common inputs result from one chord.

[0103] An example of intuitive chord-input association is the association of “space” with the 00005 chord, which is produced using only the thumb in a left handed handset; not only is this an easy chord to produce, but this also corresponds to how “space” is input with a keyboard.

[0104] OTIS optimizes input locatability by showing the LOWER, UPPER, and CAPLOCK inputs in alphabetic order, the NUMBER and NUMLOCK inputs in numeric order, and by attempting to put the most frequently used symbols first in the NUMBER, NUMLOCK, and PUNCT grids.

[0105] Related function chord-mapping is exemplified not only by the uppercase letters being produced by the same chords as their lowercase counterparts, but also by the same chord being used to switch to UPPER, and then if necessary to CAPLOCK, and then again back to LOWER. To see this, note that 12300 chord once allows a single uppercase letter to be input using the same chords as for lower case letters via the UPPER grid. Chording it twice allows a series of uppercase letters to be input using the same chords via the CAPLOCK grid. Chording it a third time returns to lowercase input with the LOWER grid. The 12305 chord behaves similarly for numbers and math symbols.

[0106] One of the benefits of a well-designed OHAI gridset application is that user dependence on the OHAI window wanes over a short period of time. When the chords are memorized, the user can ignore the OHAI window completely and focus entirely on the user application; at this point, the OHAI window can be removed from the screen, or remain for reference purposes. Thus, due to its continuous visual feedback and the small number of chords, OHAI is self-training. The self-training aspect of a well-designed OHAI gridset application is even more necessary in CHIME and HAJIME due to the large numbers of characters for Chinese and Japanese.

[0107] User Customization of OTIS

[0108] An input system which produces only alphanumeric input and forces the user to a single set of chord-text associations is of limited use, so a way to add and rearrange inputs must be provided. Indications of two kinds of customizations, extensibility and rearrangement are discussed here briefly.

[0109] Extensibility means providing additional inputs; one way of extending OTIS is to assign a user-defined successor grid to the blank selection cell for the 12340 chord in the PUNCT grid. For example, if the user uses Chinese or Japanese characters in his task, he can assign the base grid of CHIME or HAJIME to this block, thus Chinese or Japanese input is two chords away from the base grid for OTIS.

[0110] Rearrangement means reassigning inputs to chords and reflecting the reassignment in OHAI gridset application grids. Grid layouts other than those described in this application are possible. The layouts in this application are designed to minimize input location time and maximize memorizability for a “typical” user. A more specialized user—say a novelist—may decide to put the most commonly used characters on the base grid and less commonly used characters on a successor grid. Thus ‘q’, ‘x’, and ‘z’ may be swapped with the characters for period (.), comma (,) and double quote (”) Another obvious optimization involves research to determine which chords are most difficult for the user (either a “typical” user or specific users). Once this determination is made, the most frequently used letters can be associated with the easiest chords, thus sacrificing short-term locatability for long-term ease of use, an obvious application for users prone to repetitive motion disorder.

[0111] Practical Uses of OTIS

[0112] OTIS is a sample OHAI application to illustrate OHAI concepts, not an argument for OHAI replacing the keyboard for English text input in the workstation user environment. Most people who do English text input exclusively are already accustomed to the keyboard. The obvious benefits of OHAI over the keyboard for English text input occur in other user environments; for instance, the keyboard adds weight and moving parts to a hand-held computer, so for English text input to one of these devices, an OHAI device consisting of five ergonomically placed buttons on its housing (as illustrated in FIGS. 35 and 36) has advantages in terms of usability, weight, and cost over a miniature keyboard. OHAI usage in the workstation user environment for English text input is desirable only when there is a reason not to use a keyboard; such as space saving, portability, physiological and workflow benefits, and so forth.

[0113] THE OHAI INPUT SYSTEM IN GENERAL

[0114] In the OTIS example, the WOH device and the OTIS application work together in the workstation user environment to allow the user to input English text with one hand. It should be kept in mind that the WOH is a specific OHAI device and OTIS is a specific OHAI application. The two changeable components of this scheme are the OHAI device and the OHAI application. The device varies to suit the user environment, and the OHAI application varies in both form and content with the nature of the input desired. The WOH is suited to the desktop user environment and OTIS is suited to English text input. Other OHAI devices and OHAI applications are suited to other user environments and purposes.

[0115] OHAI devices and applications are components of the OHAI system, as illustrated in FIG. 15 which shows pathways for OHAI system information flow. In FIG. 15, the path from “OHAI Applications Grids” to “User's Eye” represents an OHAI application being used by the OHAI software to inform the user, via a window or chordmaps, how to produce inputs and sub-inputs. Simultaneously, the user application's window presents the user with options (e.g. a document which requires an input choice). The path from “User's Hand” to “OHAI Software” represents the user manipulating the device to choose inputs or sub-inputs. The OHAI software, using OHAI application grid information, interprets signals sent from the device and updates the contents of the window to ensure that the OHAI window keeps the user informed of both his tentative input choice (through highlighting) and his current input choices (through grid-swapping or chordmap overlaying). When an input is completely chorded, it is sent to the user application, and the process repeats.

[0116] The presence of the window and the nature of the handset eliminate two factors which work against other implementations of chord-based input systems. Instead of forcing the user to memorize associations of chords with inputs, the OHAI window informs how to achieve the desired input. By having strictly five keys, the OHAI handset eliminates the finger travel implicit in the e.g. QWERTY, Twiddler, and BAT keyboards.

[0117] DESCRIPTIONS OF OHAI APPLICATIONS

[0118] CHIME: The OHAI Chinese Input Method

[0119] The most immediately obvious benefit provided by OHAI is with input sets unsuited to the QWERTY keyboard, as with Chinese text input. More than one quarter of the world's population speak Chinese. An increasing number of these use computers in their day to day lives. It is desirable that these users be able to input Chinese characters to user applications.

[0120] The Chinese Input Problem

[0121] Depending on the authority cited, the number of Chinese characters (hanzi) ranges from the thousands to the tens of thousands. It is generally agreed that the number in modern usage does not exceed ten thousand. Even with only ten thousand hanzi, it is a technical challenge to minimize user effort in selecting them to be input to user applications.

[0122] In addition to the sheer number of hanzi, each is internally complex, being comprised of from one to twenty-nine strokes.

[0123] Another factor contributing to the complexity of devising an easy to use input method for ten thousand internally complex hanzi is the lack of a single organizing principle. Each hanzi has many graphical attributes which are used to classify and organize them in e.g. dictionaries; these organizing and classifying attributes include constituent strokes, constituent parts, stroke order, stroke count, and “radicals”, which are a finite set of cognitively primary constituent parts. Although some very clever input methods have been devised utilizing graphical attributes of hanzi, these force the user to memorize a tremendous amount of information, take essentially the same amount of time to input hanzi as to write them by hand, or involve specialized expensive hardware, and so have failed to gain popular usage.

[0124] The problem is to devise a system for the selection of hanzi which takes advantage of preexisting knowledge about hanzi, minimizes additional required learning, and, once learned, takes less time and effort to use than the actual writing process. If additional hardware is required, it should be inexpensive and useful for additional purposes.

[0125] Chinese Input Solution Strategy

[0126] The input methods which demand the least effort on the part of the user take advantage of the fact that, although there are of the order of ten thousand hanzi, the number of pronunciations is an order of magnitude smaller (in Mandarin there are only one thousand two hundred and seventy nine pronunciations). These input methods allow the user to narrow the selection of hanzi by first selecting the pronunciation and then selecting from among the range of hanzi with that pronunciation. The most popular methods implement the pronunciation specification phase in this process using the pinyin pronunciation representation system, so before showing how these systems work, a summary of pinyin is apposite.

[0127] Précis of Pinyin Romanization

[0128] Pinyin, the most widespread method of representing pronunciation, is taught in elementary schools in all countries which teach Chinese as part of the standard curriculum. It uses the Roman alphabet and either diacriticals or numerals to represent Mandarin pronunciations.

[0129] Pinyin representations optionally start with one of 21 consonant sounds:

[0130] b, p, m, f, d, t, n, l, c, s, ch, sh, r, j, q, x, g, k, h

[0131] In CHIME the option not to have an initial consonant sound is itself treated as an initial and is denoted ‘ø’. Regardless of how they start, all pinyin have one of nineteen vowel combinations:

[0132] a, o, e, -i, er, ai, ei, ao, ou, i, ia, ie, iou, U, ua, uo, uai, uei, uë

[0133] Although they use the same letter, ‘-i’ and ‘i’ sound different; ‘-i’ resembles the vowel sound in English “is” and ‘i’ resembles the vowel sound in English “she”. If there is no optional consonant prior to the vowel cluster, multi-letter vowel clusters starting with ‘i’ or ‘u’ are substitute ‘y’ and ‘w’ respectively in written pinyin. Hence, “ya-”, “yao”, “you”, “yo-”, “ye”, and “wa-”, “wo”, “wai”, “we-”.

[0134] In addition to the optional initial consonant sound and the mandatory vowel cluster, there are two optional endings:

[0135] -n, -ng

[0136] In CHIME the option not to have final is itself treated as a final and is denoted ‘ø’.

[0137] Most pinyin have one of four tones: the first tone is high and level, the second tone is short and rising, the third tone is long and falls then rises slightly. The fourth tone is short and falling. The absence of tone is sometimes referred to as the neutral tone and is short and neither rises nor falls. Each of the non-neutral tones is indicated by a numeral following the letters, or by a diacritical mark over the principle vowel in the vowel cluster:

[0138] ma1, ma2, ma3, ma4, ma

[0139] or

[0140] ma, ma, ma, ma, ma

[0141] The neutral tone is sometimes indicated by a 0 or ø following the syllable or by a ° or over the principal vowel. The absence of a diacritical or following number is also sometimes used to indicate that the tone is unknown or not relevant.

[0142] It is useful to distinguish between syllables and pinyin; for the purposes of this section, pinyin denotes complete pronunciations including tone, and syllable denotes the pronunciation without tone. For example, “ma” is a syllable, “ma1” and “mã” are pinyin, and □ {“mother”} is a hanzi.

[0143]
FIG. 16 shows all of the combinations of initials, vowel clusters, and finals that comprise Mandarin syllables. Note that not all possible combinations of initials, vowel clusters, and finals actually exist.

[0144]
FIG. 17 shows how many of the pinyin derived from the syllables in FIG. 16 exist in the dictionary we used as reference. Note that not all possible pinyin actually exist.

[0145] Using Pinyin to Select a Range of Hanzi

[0146] To illustrate pinyin-based input methods, consider input of the single Chinese hanzi □ {“shi4” “is”}. As shown in FIG. 18, □ 30 is one of some forty two hanzi with the same pinyin.

[0147] To input □, the user begins by choosing “s-h-i-4”. The software implementing the input method then uses some visual means to present the hanzi and allow the user to select among them.

[0148] The means for “choosing” pinyin and the visual means for presenting the hanzi once the pinyin has been specified vary widely among implementations of Chinese input methods.

[0149] Example Keyboard Implementation

[0150] One of the most popular pinyin-based input methods is TwinBridge Software Corporation's Chinese Partner®, which translates keyboard keystrokes to hanzi. Although Chinese Partner is a very popular and very useful application, the nature of the keyboard and the input task render touch-typing impossible. To see this, take the task of selecting □ for input.

[0151] The user indicates the pinyin by typing “s-h-i-4” using the corresponding keys on the keyboard, then looking at the TwinBridge prompt window (FIG. 19) to see if it is one of the ten hanzi displayed. If the hanzi is displayed, one of ten function keys is pressed to indicate which hanzi is to be sent to the user application. If the hanzi is not displayed on the first prompt screen, then the arrow keys must be used to display a different set of ten hanzi with the same pronunciation within the prompt window (FIG. 20). When the hanzi does appear, one of the ten function keys is used to select it.

[0152] The process is similar for all hanzi. Using Chinese Partner, the average user types a minimum of five keys to specify a single hanzi, an average of seven, and, assuming he spots the hanzi on the first pass, a maximum of twelve. If he misses the hanzi in the prompt window as he is using the arrow keys, he must use the arrow keys to go back through previous screens to locate the hanzi, thus adding keystrokes. Missing the hanzi on the first pass through the prompt screens is fairly common due to the constant need to switch visual focus and attention among the keyboard, the prompt screen, and the user application.

[0153] Chinese Partner is among the most efficient Chinese text input systems for the keyboard, its unwieldiness is due only to the nature of the task to which it is set, namely attempting to use for Chinese input a device intended originally to slow down the input of English text.

[0154] OHAI Implementation

[0155] CHIME—the OHAI CHinese Input MEthod—improves upon the basic two phase strategy used by Chinese Partner: the user first specifics pinyin, then selects among the hanzi for that pinyin. CHIME achieves the same goal with less user effort by modifying the Chinese Partner's strategy to take advantage of OHAI's features. This section shows how to use CHIME and then presents some design considerations and advantages, as well as other uses for the same interface.

[0156] Using CHIME for Chinese Input

[0157] OVERVIEW OF CHIME

[0158] Like OTIS, CHIME consists of grids which are swapped in and out of the OHAI window. These grids lead the user through the process of selecting hanzi to be input to user applications. All hanzi can be chorded in from two to five chords.

[0159] Like Chinese Partner, the process of selecting a hanzi occurs in two phases: the pinyin selection phase and the hanzi selection phase. The pinyin selection phase has three stages or junctures, each defined by what is displayed in the CHIME grid for that juncture. These are: initial (or base) juncture, syllable juncture, and pinyin juncture. CHIME pinyin phase junctures are represented in FIG. 22 and FIG. 23. The hanzi selection phase consists of one juncture for the vast majority of cases, and, only when the user does not know a hanzi's tone, more than two; where necessary hanzi junctures are distinguished by number.

[0160] At the initial selection juncture, the OHAI grid displays all pinyin initials including “ø/w/y” (to allow the user to indicate that there is no initial consonant for the desired hanzi). Note that some selection cells contain more than one initial, and some contain two-letter initials. To input □, the user begins by chording 02345, which is associated with the pinyin initial ‘sh-’. After completing the initial juncture, the user enters the syllable selection juncture.

[0161] At the syllable selection juncture, each selection cell is associated with a vowel cluster. The selection cells display all existing Mandarin syllables which begin with the previously selected initial and the associated vowel cluster. The syllable “shi” is displayed in the selection cell associating the chord 00300 with the ‘-i’ and ‘-er’ vowel clusters. To demonstrate what is meant by “display existing syllables”, note that for the vowel cluster ‘-ue-’ there is no “shueng” in Mandarin but there are syllables pronounced “shui” and “shun”, so these are displayed in the selection cell associating the chord 10040 with the vowel cluster ‘-ue-’. Note also that there is no Mandarin syllable “shiou” or “shiong”, so the selection cell associating the chord 02305 with the vowel cluster ‘-iou-’ is empty. At the syllable selection juncture, the user may be choosing as many as three different syllables, depending on the finals which follow the initial and vowel cluster. These are narrowed down to one in the pinyin selection juncture.

[0162] At the pinyin selection juncture, each selection cell is associated with a combination of final and tone. The selection cells display all existing pinyin based on the previously selected syllable set. The columns in grids at this juncture are associated with finals and the rows are associated with tones. To enhance locatability, the second, third and fourth columns are used (enabling the user to start a left-to-right visual scan immediately after the name block) rather than the first, second and third. The first final is ‘ø’ (indicating that there is no final) followed by ‘-n’ and ‘-ng’. The first through fourth tones are taken in order, followed by an “any tone” option (in case the tone is unknown), and then the neutral tone (which is rarely the only tone of a hanzi). The pinyin “s h i” is displayed in the second column (associated with the ‘-ø’ final), and the fourth row (associated with the fourth tone). The chord for this combination of final and tone is 02305. Once the pinyin is selected, its hanzi are displayed at the hanzi juncture.

[0163] Even if the hanzi carries no tone in the context in which it is being input, it will usually have an additional, and primary, toned pronunciation, which will often be used to locate it in CHIME. For instance the hanzi “I suppose” used at the end of sentences is pronounced with both neutral and first tone and so may be located in grids selected with first, any, or neutral tone.

[0164] At the hanzi selection juncture, each selection cell is associated with a single hanzi pronounced with the previously selected pinyin. As many as possible are displayed in the OHAI window. If there are more than thirty hanzi, twenty-nine are displayed and the chord 12340 causes the OHAI window to display the remaining hanzi (this is the second hanzi juncture). When the user chords for one of these, OHAI sends the hanzi to the current user application window and returns to the initial screen to wait for another hanzi to be selected.

[0165] All pinyin are selected using a sequence of at most three chords: the first chord in the sequence selects an initial, the second chord selects a vowel cluster, the third selects the tone and final. At each juncture in the sequence, a chord always selects the same choice for that part of the pinyin: for example, the 02005 chord at the first juncture of the sequence always selects the initial ‘l-’, at the second juncture always selects the vowel cluster ‘-i-’ (provided it exists for the previously selected initial), and at the third juncture always selects for the third tone with no final (also providing they exist for the previously chorded initial and vowel cluster). Only those pinyin syllables which exist and combine initial, vowel, final and tone are shown as the third chord choices.

[0166]
FIG. 23 has a table showing which chords select which pinyin components at each of the three junctures in the pinyin selection process. Once a pinyin is selected, the hanzi for it are always presented in the order of the number of strokes required to produce the hanzi with pen and paper (a very familiar organizing principle easily used by anyone familiar with hanzi). This takes advantage of a tendency for the most commonly used hanzi to be those with fewest strokes, and also those which are most easily identified, thus speeding locatability.

EXAMPLES

[0167] Because very few pinyin have more than thirty hanzi, approximately 99% of hanzi are selected in four or fewer chords. 95% are selectable in exactly four chords, 4% require less than four chords, and 1% require five chords. Since no pinyin has more than fifty nine hanzi, none requires more than five chords. Examples of hanzi selectable in four, five, two and three chords follow:

[0168] Typical Hanzi is Four Chords

[0169] Four chords are sufficient to send well over 95% of all hanzi to user applications. FIG. 24 illustrates the four chord sequence needed to input □ (“shi”). Starting at the base grid, the user chords 02345 to select the initial ‘sh-’. When this initial is selected, the second grid, which shows all syllables starting with the initial ‘sh-’, allows the user to select the syllable ‘shi’ with the chord 00300. The resulting third grid allows the user to select the complete pinyin with the 02305 chord. The fourth grid displays the first twenty nine of the forty two hanzi pronounced s h i; the user then chords 10345 to select □, sending it to the user application and causing the OHAI window to display the base grid again in preparation for the next hanzi.

[0170] The chord sequence 02345003000230510345 is all that is necessary to send the hanzi □ (“shi”) to a user application such as a word processor or spreadsheet. Note that four chords are less than the number of strokes (nine) required to produce the hanzi on paper.

Examples for Five Chord Hanzi

[0171] In those cases where there are more than thirty hanzi for a pinyin, the first twenty nine are displayed at the fourth juncture and those thirty and greater are displayed in a fifth. The hanzi □ (shi4, “vow or pledge”), indicated by 31 in FIG. 18, is an example. The ellipsis in FIG. 25 indicates that there are more than thirty hanzi pronounced “s h i”. To access these, the user first uses the same first three chords (023450030002305) as □ and then, at the first hanzi juncture, chords 12340 to access the fifth chord map. Finally 02045 chords □. Here, the five chords (0234500300023051234002045) are less than the fourteen strokes required to hand-write the hanzi. Other pinyin for which there are 5 chord hanzi are:

[0172] yi4, yu4, bi4, li4, zhi4, jil, ji4, xil

Example for Two Chord Hanzi

[0173] If there is only one hanzi that exists for a particular initial and a vowel cluster, then that hanzi is displayed at the syllable selection juncture, allowing the user to select it in only two chords. FIG. 26 shows the CHIME grids and chording sequence for inputting the hanzi □ (gei3 “give”) in only two chords. Starting at the CHIME base grid, the user chords 10005 to select the initial ‘g-’. Since gei3 is the only pinyin with the initial ‘g-’ and the vowel cluster ‘-ei-’, and □ is the only hanzi for this pinyin, □ is displayed in the grid block associated with the vowel cluster ‘-ei-’. The user chords 10340 to select it. Two chords (1000510340) compare very favorably with the twelve strokes for □. Some other hanzi that are selected in two chords, and the number of strokes necessary to produce them, are shown in FIG. 27.

[0174]
FIG. 22 can be used to determine which sequences of two chords input each of these.

Examples for Three Chord Hanzi

[0175] If there is only one hanzi for a particular pinyin, but there are other pinyin that begin with its initial and vowel cluster, then that hanzi will be displayed at the hanzi selection juncture, allowing the user to select it in only three chords. FIG. 28 shows the CHIME grids and chording sequence for inputting the hanzi (□ bail “pinch off”) in only three chords. Starting at the CHIME base grid, the user chords 00040 to select the initial ‘b-’ and then 00305 to select the syllable ‘bai’.

[0176] Since bail has only one hanzi, it is displayed in the grid block associated with the first tone and no final. The user chords 00005 to select it. The three chords (000400030500005) here compare favorably with the twelve strokes necessary to write □. Other three chord hanzi and stroke counts are as shown in FIG. 29.

[0177]
FIG. 22 can be used to determine which sequences of three chords input each of these.

[0178] The Design of CHIME

[0179] CHIME's sine qua non is to enable hanzi input. If it is to become commercially viable, equally important is that it minimize user effort in the process. No less desirable, but of secondary importance, is that it provides means for customization of other input functions. This section presents an overview of CHIME's user effort minimization, enumerates some of the optimizations that contribute to CHIME's efficiency, and then shows how CHIME can be customized.

[0180] Minimizing Effort

[0181] CHIME minimizes both long term and day-to-day user effort.

[0182] Long Term User Effort Minimization

[0183] CHIME minimizes long term user effort through a self-teaching design which diminishes the necessity to look at the OHAI window over time. This self-teaching is achieved by consistency of both sequence and choice; i.e., the same choice at the same juncture in the process always produces the same result. Combining sequence and choice minimization with consistency enables the user to memorize chord/input associations by using them, not through any special effort. The result is that the user capitalizes on his day-to-day effort: in a small amount of time, he no longer has to look at the OHAI window to select pinyin; and in an only slightly longer amount of time he does not have to look at the window for the hanzi he uses most.

[0184] Day-to-Day User Effort Minimization

[0185] Day-to-day effort minimization means less effort is needed to produce hanzi with CHIME than with any other method, including writing. This is achieved through the basic design of CHIME, which accomplishes the following:

[0186] 1. Minimization of the number of steps in the chording sequence to produce a hanzi.

[0187] 2. Minimization of the number of choices necessary at each step in the chording sequence.

[0188] 3. Utilization of an order among the choices at each step that facilitates location of the desired input or sub-input.

[0189] These qualities of CHIME are accomplished by optimizations listed hereafter.

[0190] Optimizations

[0191] If CHIME were to simply enable the user to select one pinyin letter after another, and then select from the hanzi for that syllable, it would take only slightly less effort than keyboard-based methods. CHIME makes a number of optimizations that enhance its suitability to the task of Chinese input.

[0192] Mutually Exclusive Initials, SemiVowels, and Vowel Clusters

[0193] Combining choices wherever possible is an optimization which frees chords, and thus allows them to be used for other purposes. Note that there are twenty one initial consonants, which together with the option not to have an initial consonant, gives twenty two choices for starting a pinyin. Even though the twenty two choices fit comfortably within the thirty one selection cells in an OHAI grid, it is desirable to take up as few as possible, especially in the base grid, in order to leave the remainder free for other uses. Observations about Mandarin syllables (FIG. 16) enable us to free up three chord blocks for the base (initial selection) grid, and one chord block in the second (vowel selection) grid.

[0194] The first observation concerns two groups of initials: (j, q, x) and (z, c, s). The initials (j, q, x) are mutually exclusive with (z, c, s) in what follows them; (j, q, x) precede the long i and the umlaut ü; (z, c, s) precede the short -i, a, e, o, and u. This can be seen graphically in the greyed areas in FIG. 16. These gaps and the phonetic similarity of the pairs (j, z), (q, c) and (x, s) allow CHIME to combine the pairs/columns and thus decrease by three the number of chords for the initial consonant choice.

[0195] There are three mutually exclusive ways to represent syllables which do not start with a consonant. For vowel clusters starting with ‘a’, ‘o’, or ‘e’, the syllable starts with the vowel. For vowel clusters starting with ‘i’, the pinyin system substitutes ‘y’. For vowel clusters starting with ‘u’, the pinyin system substitutes ‘w’. These three mutually exclusive choices are combined in the selection cell associating the chord 00005 with the absence of an initial consonant.

[0196] A similar observation allows CHIME to free a chord block in the vowel selection phase: Short ‘-i’ never occurs without a preceding consonant and ‘-er’ never occurs with one, so the same second chord can select both. FIG. 21 is the result of making these collapses in the Mandarin syllable chart in FIG. 16.

[0197] Initials & Vowel Clusters Rather than Letters

[0198] Selecting sequences of sub-inputs rather than individual sub-inputs wherever possible is an optimization which reduces junctures thus enabling faster input. In CHIME this optimization manifests as the ability to select two-letter initials (e.g. ‘sh-’), entire vowel clusters (e.g. ‘-iao’), and tone and final in one chord. The alternative, in this case selecting one letter at a time, is possible with OHAI, but choosing this and other optimizations enables CHIME to improve upon previous methods in terms of usability and speed.

[0199] Using this strategy, each CHIME juncture saves user effort: the base juncture saves as many as one chord through the inclusion of two-letter initials in the base grid; the syllable juncture saves as many as two chords through the presentation of all Mandarin vowel clusters; The pinyin juncture saves as many as two chords through combining tone and final. Since the user needs fewer chords to produce a hanzi, the input process is speeded up.

[0200] Minimization of Choices at each Juncture

[0201] Memorizability is enhanced by minimizing the total number of chord/input associations, the total number of junctures, and the number of associations within each juncture. FIG. 23 shows that the number of associations at the base juncture is twenty four, the number of associations at the syllable juncture is twenty one, at the pinyin juncture, eighteen. Thus sixty four facts are memorized to produce one thousand two hundred and seventy nine pinyin. This number allows the whole pinyin selection process to be memorized much more easily.

[0202] Consistency of Choices at each Juncture

[0203] When the user looks at the OHAI window, he always sees the same choices in the same place at each juncture. This enhances locatability by ultimately eliminating the necessity to scan at each juncture. As the sixty four associations of chord with pinyin part become familiar, the user's eye gets trained to look at the selection cell directly. This contributes to speeding the input selection process.

[0204] Any Tones Selectability

[0205] Sometimes native speakers are not sure of the tone for a particular pinyin, so CHIME includes an “any tone” option at the pinyin juncture. When this selection is chorded, all hanzi for all tones are displayed in as many subsequent grids as necessary to accommodate them. FIG. 30 shows the grids displayed after selecting the any tone option for “shi”. The grids display all hanzi in all tones for the syllable “shi” and are arranged in stroke count order. The any tone grids are the sole exception to no hanzi requiring more than five chords.

[0206] Preview of Next Grid

[0207]
FIG. 28 shows a grid that is a mix of pinyin and hanzi. When there are four or fewer hanzi associated with a particular pinyin, CHIME displays them instead of the pinyin. This has the advantage of indicating which hanzi will appear on the next grid and in which order. If the hanzi is used often, its position in the chord block at the previous juncture comes to indicate the required chord without having to look at the next grid.

[0208] Changed Mind Optimizations

[0209] Since there are many free selection cells in each of the first three junctures, CHIME allows SPACE, 0-9, PUNCT, and CANCEL in them all so that the user can at any point in the pinyin selection process enter a space, number, math symbol, punctuation, or go back to the base grid. This means the user does not have to go back to the base grid if he changes his mind while in the process of selecting a pinyin.

[0210] Display Only What Exists

[0211] So not to distract the user, at each juncture in the Chinese text input process, CHIME displays only those syllables, pinyin, and hanzi that exist, and not those which may exist (for instance, as a result of combining the selected initial with all of the vowel clusters in the second grid, or a selected syllable with all possible tones), this minimizes visual distraction thus enabling the user to more quickly locate selections.

[0212] Intuitive Selection Cell Contents

[0213] Locating inputs and sub inputs is facilitated by displaying selection cell contents which tell the user as much as possible as clearly as possible what will happen when the chord associated with the selection cell is chorded. In chording, the sequence of selection cells is “sh”, “shi”, then “s h i” rather than “sh”, “-i”, then “ø(4)”, which are the actual meaning of the chords. The user thus sees what he expects to input rather than it's bits and pieces.

Customizability Example

[0214] The presentation of a minimal number of choices, especially for the first and last sequential chords, dovetails with customization goals of CHIME. Examples of customization functions are: user defined symbols, selection of pinyin input instead of hanzi, and selecting multi-hanzi phrases which begin with the selected hanzi. Customization entry points are inserted at the first and last junctures in CHIME; between these junctures, the user is in the process of selecting a hanzi, which must either not have started or be complete before customization functions are appropriate.

[0215] One kind of customization, extensibility, consists of enabling the user to do something other than selecting a pinyin initial at the first CHIME grid. This is achieved by using free grid blocks for the extension functionality. If, for instance, the user wanted to be able to switch between Chinese and English input, he could integrate OTIS into CHIME by using one of the first juncture selection cells to select OTIS, and a free OTIS selection cell to return to CHIME. One scheme for this is given in FIG. 31.

[0216] For user convenience each OHAI gridset applications base grid is relabeled with the name of the language input. To complete the integration, all that is needed is a way to start OTIS from CHIME and a way to start CHIME from OTIS. To make OTIS accessible from CHIME, all that is needed is to use one of the free chord blocks in the base grid to reference the base grid of OTIS. To make CHIME accessible from OTIS is a little trickier because all of the cells are used. In fact, there is only one free cell in all the OTIS grids. Not wanting to do a lot of rearranging, and accepting the necessity to chord into the PUNCT grid in order to access CHIME we use this free cell to reference CHIME's base grid. So, to start OTIS from CHIME the user chords 12000. And to return to CHIME from OTIS, he chords 1234012340. Both of these are easily remembered.

[0217] Advantages of CHIME Over Keyboard-Based Methods

[0218] TwinBridge displays only ten hanzi at a time and forces the user to use keystrokes to swap out other sets of ten which takes time. CHIME displays all of the available hanzi in one of two grids. Since the OHAI handset is used, and the OHAI window occupies the same display as the application window, the user does less visual focus switching. The user does not have to backtrack through prompt screens. For only 1% of hanzi is it possible to miss on the first pass.

[0219] Other uses of CHIME

[0220] The interface for selecting hanzi for input can be used for other applications. One example of an application that uses the same interface is a Chinese dictionary. Instead of sending input, the dictionary application can display a definition of the selected character. Another possible use is as a teaching and translation aid: for non-fluent speakers of these languages, the same software can inform the user of the meaning of a selected hanzi. In lookup mode, OHAI displays translations for the chorded hanzi, and only sends the hanzi to the user application window if the chord is re-selected.

[0221] TECHNICAL: UNDER THE HOOD OF OHAI

[0222] OHAI Hardware and Firmware

[0223] The WOH is one example of the OHAI hardware. OHAI hardware can take any form as long as it sends chord information to the OHAI software. OHAI Firmware is a program that resides on the chip in the hardware that translates chord presses into interpretable signals and sends them to the OHAI software on the user's computer.

[0224] Firmware Description

[0225] All the firmware does is monitor the state of five switch inputs corresponding to the fingers holding the OHAI device. When the aggregate state of these five inputs changes, the firmware causes the device to send the corresponding chord to the computer on which the OHAI software resides. FIG. 33 is a flowchart showing one method of implementing the firmware. Note that this design guarantees that no duplicate chords are sent in sequence, and that one and only one NULL chord is sent when a chord is a completely released.

[0226] The flowchart in FIG. 33 implements the expression

16 (F4)+8 (F3)+4 (F2)+2 (F1)+T

[0227] where F is a finger press and T is the thumb press (Lower Fx is closer to the thumb). This expression ensures that the integer associated with a chord is unique for each combination of finger presses.

[0228] The Chord Accumulator, as shown in FIG. 33, is used so that the “Send Chord” procedure only sends chords that are complete.

[0229] Hardware Forms

[0230] In a user environment where both right and left hand users use the same computer, it is advisable to have a form of the OHAI device that can be used easily in either hand. A device which meets these requirements is shown in FIG. 39.

[0231] The workstation OHAI handset illustrated in FIG. 39 resembles a joystick and has a base and a key carrier extending from the base. The key carrier includes a top surface having a single pushbutton switch 50 which is depressible by a thumb of a user, and a set of four pushbutton switches 51, 52, 53, 54 extending along a front surface of the carrier facing away from the user. The siting of these pushbutton switches enables the device to be grasped and used by a right hand or a left hand.

[0232] The simplicity of requirements for the OHAI hardware make possible a plethora of designs, each suitable to its particular user environment or user preferences. For instance, one of the criteria which define user environments is the degree of mobility required. A moderately mobile user uses a laptop. A very mobile user requires a very small computer that can be set up and taken down in very short periods of time; wearable or palmtop computers are suited to this environment.

[0233] A form of the OHAI device suitable for laptops is shown in FIG. 35 and FIG. 36. The OHAI laptop device consists of one or two sets of five buttons 40, 41 and 43, 44 ergonomically placed on the housing of a laptop computer. Although either set of buttons would serve, having two allows the user to alternate between them to eliminate hand fatigue caused by too much time in the same position. Similar button configurations can be placed on armtop, palmtop and wearable computers.

[0234] In FIGS. 35 and 36, button 43 is the topmost of a set of buttons for use by a left hand on the monitor portion of the laptop computer. Button 43 is for the thumb of the left hand. Buttons 44 are for use by the fingers.

[0235] A set of buttons 40 and 41 are the thumb and finger buttons respectively ergonomically and alternatively located on the laptop. The little finger depresses the leftmost button 45.

[0236] For maximum comfort, the user can use either the front set (40, 41) or the back set (43, 44) of buttons at any time.

[0237] The OHAI Glove (FIG. 37) is a suitable version of the OHAI device for wearable (especially military field-use) computers. It consists of five sensors integrated into the finger tips of a glove, the OHAI firmware and a method for sending chords to a wearable computer when the fingers are pressed. If the wearable computer has a headset (FIG. 38), the interaction of the glove and the OHAI software and the OHAI window in the headset display is identical to the interaction of the WOH, the OHAI software, and a monitor screen.

[0238]
FIG. 39 shows an example of chords corresponding to letter for use with non-display applications.

[0239] There is a specialized OHAI interface for audio/vocal interfaces such as might be used in military field applications requiring secure silent communications. In these cases the computer is connected to a radio or other means of communication. Since silent communication is required, FIG. 41 gives a schema for such secure silent communication.

[0240] IMPLEMENTING OHAI APPLICATION TYPES

[0241] All OHAI applications include software which receives chording information from the OHAI hardware. The software is different for each of the three types of OHAI applications.

[0242] OHAI Gridset Applications

[0243] CHIME and OTIS are examples of OHAI gridset applications.

[0244] Since there are many different OHAI gridset applications, and they are modifiable by the user, the complexity is managed in data files. Since the data files are separate from the OHAI software, they can be changed without any danger of breaking OHAI. Each of these will be described in sufficient detail that a reasonably competent programmer can implement the entire system'. There are so many ways to optimize the system that these are left up to individual implementers. Rather than present a comprehensive theory of implementation, we make extensive use of OTIS as an example. Since the data files determine what OHAI does, this section starts with a description of an OHAI gridset application file.

[0245] DATA FILES FOR OHAI GRIDSET APPLICATIONS

[0246] Overview of OHAI Gridset Application Files

[0247] OHAI gridset application behavior is completely specified by ORAI gridset application files. OHAI gridset application files contain information which specifies the selection cell contents, the user application input and the selection cell successor grids.

[0248] For instance, the OTIS base grid (FIG. 5) selection cell for the chord displays “space”, causes a space character (ASCII 32) to be sent to the User Application, and has the base grid itself as a successor.

[0249] Organization of the OHAI Gridset Application File

[0250]
FIG. 32 shows the structure of the file for the OTIS OHAI gridset application. The file consists of entries for each selection cell. Each entry consists of three fields; the fields are named display, input and successor.

[0251] The display field indicates what is displayed in the selection cell.

[0252] The input field indicates what is sent to user applications.

[0253] The successor field indicates the successor grid. The NULL constant is used to indicate nothing is displayed if it is in the display field, that nothing is sent to the user application if it is in the input field. For inputs that cannot be displayed, such as space and backspace, a backslash (\) is used to precede the corresponding ASCII code. If backslash is the input, two backslashes are used as the input field. There is always a name of a grid in the successor field. There are thirty two sequentially arranged entries for every grid in an OHAI application. The grid field always refers to one of these grid groups.

[0254] An Implementation of OHAI Gridset Application Files

[0255] One way of implementing OHAI gridset application files is to use tabs to delimit the fields in each entry and carriage returns to delimit entries. Using this implementation, lines of the OTIS file would look like the following:

1LOWERNULLNULLspace\032LOWERaaLOWERbbLOWER. . .UPPERNULLNULLSpace\032LOWERAALOWERBBLOWER. . .

[0256] The file has 192 lines and the 32 lines for each grid group arc arranged so that the first entry is for the title block, the second entry is for the first chord (00005), the third entry is for the second chord (00040), etc. For all but the base grid, which must be first, the grid groups can be in any order.

[0257] THE OHAI SOFTWARE FOR OHAI GRIDSET APPLICATIONS

[0258]
FIG. 34 is a flowchart showing how the OHAI software uses OHAI gridset application data files to process input from the OHAI hardware to send inputs to user applications. It might be helpful to think of the output of the OHAI software as the input to user applications.

[0259] The first thing the software has to do is bring up the contents of the OHAI window. The initial contents are determined by the first grid group in the file. All of the display fields in the first grid group entries are displayed in the OHAI window top-to-bottom and left-to-right so that the sequence in the file matches the sequence in the window.

[0260] After initializing the OHAI Window, the OHAI software sets a variable that keeps track of the second most recent (the “last” chord) chord received from the OHAI device to the NULL chord.

[0261] Once these initializations are complete, the main loop of the software is entered. The first thing that the loop does is wait for a chord to be received from the device. Once it receives a chord into a variable that keeps track of the most recent chord (the “current” chord) there are three possibilities:

[0262] 1. The current is chord is the NULL chord

[0263] 2. The current chord is not NULL and the last chord is NULL

[0264] 3. The current chord is not NULL and the last chord is not NULL

[0265] Case (1) corresponds to the user pressing and releasing a chord. Since the firmware cannot send two instances of the same chord in a sequence and sends the NULL chord when all switches are released, case (1) can only occur if the last chord was not NULL. This means that the user pressed and released a chord thus selecting it; this pressed chord was recorded as the last chord and releasing it caused the current chord to become NULL. The first action the software takes to process a selected chord is to remove the highlight for the last chord (it is no longer tentative). After removing the highlight, any input associated with the chord is sent to the user application (the input field is not NULL). If the successor grid for the selected chord/input is different from the current grid, the current grid is set to the successor, it is located in the file, and the OHAI window updated for it. After processing the chord's display, input, and successor grid, the software overwrites the last chord with the current chord so processing can resume again by waiting for a chord from the device.

[0266] If the current chord is not NULL (cases 2 and 3), that means that the user is either starting or in the midst of a sequence of tentative chords.

[0267] Case (2) corresponds to a user pressing a chord but not releasing it after having already released the previous chord. In case (2), the last chord received from the device was NULL indicating that the last chord was completely processed (or that this is the very first chord received by the software); this means that the current chord is the first (and possibly last) in a sequence of tentative chords. The only thing that the software does for the first tentative selection is highlight the selection cell for the chord in the OHAI window. After updating the OHAI window, the software overwrites the last chord with the current chord so processing can resume again by waiting for a chord from the device.

[0268] Case (3) corresponds to the user pressing a chord after a different chord with no intervening release. This is the case when he changes his mind, or when he presses one finger at a time to complete a chord before releasing; each time he adds a finger is a different tentative chord. With each change, the software removes the highlight for the last chord and highlights the selection cell for the current chord. After updating the OHAI window, the software overwrites the last chord with the current chord so processing can resume again by waiting for a chord from the device. The loop which processes chords continues indefinitely.

[0269] While the invention has been described with reference to specific embodiments, modifications and variations of the invention may be constructed without departing from the scope of the invention, which is defined in the following claims.

	Number	Date	Country
Parent	09031663	Feb 1998	US
Child	09764747	Jan 2001	US

OHAI technology user interface

Information

Publication Number

Date Filed

Date Published

Inventors

CPC

US Classifications

International Classifications

Abstract

Description

Claims

BACKGROUND OF THE INVENTION

Continuation in Parts (1)