Patent Application
20060279559
The present invention relates to electronic handwriting equipment, and more particularly to an apparatus for handwriting recognition having a handwriting user interface operable in at least a first mode and a second mode, which are selectable by a control panel. The invention also relates to an associated method for handwriting recognition.
Examples of electronic handwriting equipment include personal assistants (PDAs), hand-held computers (palmtops) and mobile terminals for telecommunication (mobile phones). These apparatuses have in common that they make use of a stylus and a touch-sensitive display screen, a solution that offers improved usability and flexibility compared to conventional user interfaces with a keypad or keyboard.
In an apparatus that accepts user input by way of a stylus and a touch-sensitive display screen, the stylus normally plays two roles; sometimes it functions like a normal pen (“logical pen”) for writing and sometimes like a control device (“logical mouse”) for controlling the user interface.
A general problem with electronic handwriting equipment is that there are some important design factors that are more or less in conflict with each other. On the one hand, it is desired to provide a handwriting input area on the touch-sensitive display screen which is as large as possible, to the benefit of the user. On the other hand, there is a strong trend towards smaller and smaller apparatus sizes for portable devices in general, including electronic handwriting equipment. Obviously, a smaller apparatus size affects the maximum space that is available for the touch-sensitive display screen and therefore also limits the size of the handwriting input area. Consequently, designing electronic handwriting equipment is often a trade-off between maximizing the size of the handwriting input area and minimizing the total apparatus size.
Often, the user interface of an electronic handwriting apparatus is operable in different modes, each related to a respective aspect of handwriting. For instance, it is very common in electronic handwriting equipment to provide one mode for each of a number of different symbol sets or character sets. The accuracy of the handwriting recognition is generally better for restricted symbol sets with a smaller number of symbols than for symbol sets with a larger number of symbols. One well-known example is the close similarity between the lower-case Latin “l” and the Arabic numeral “1”. By dividing all the various possible symbols, that a user may want to input by electronic handwriting, into different symbol sets and assigning each symbol set to a respective mode, the user may select which mode that is to be the currently active one and thereby improve the accuracy and speed of his electronic handwriting, since whatever symbol he writes will be interpreted against only the symbols that are included in the symbol set assigned to the currently active mode. A common disposition of symbol sets is to include Latin letters in one or two symbol sets (upper case and lower case), whereas numeric symbols (e.g. Arabic numerals and mathematic signs) are included in another symbol set. Additionally, non-western symbols such as Chinese characters may be included in yet other symbol sets.
The prior art generally suggests two different approaches of controlling which symbol set to use for matching against a hand-written input in handwriting recognition:
1. Dividing the handwriting input area into different sub areas, where each sub area represents a certain symbol set. These sub areas may be formed as limited-sized input boxes, each of which having a size and shape suitable for about one hand-written symbol. In other words, the user will write one hand-written symbol at a time within such an input box, and the handwriting recognition engine of the apparatus will apply the symbol set associated with this input box. A drawback with such input boxes is that they constantly occupy parts of the touch-sensitive display screen and therefore prevent these parts from being used for other purposes, such as presenting information or control objects in the user interface. This drawback is particularly pronounced if several symbol sets, and thus several input boxes, are available.
Alternatively, like in the P800/P900/P910 series of PDA-type mobile phones from Sony Ericsson, the entire handwriting input area may be logically divided so that hand-written input in the upper region of the handwriting input area is interpreted by the handwriting recognition engine as numeric input, whereas hand-written input along a horizontal center region of the handwriting input area is interpreted as upper-case Latin letters, and hand-written input in the lower region of the handwriting input area is interpreted as lower-case Latin letters. A drawback with this solution is that the meaning of each region is not automatically intuitive to the user. Particularly after a period of inactivity, the user may find it difficult to remember whether e.g. upper-case letters are to be written in the lower region, center region or upper region. Also, the solution is non-flexible in the sense that the user interface cannot be easily expanded from three different available symbol sets to e.g. four.
2. Setting the current mode by performing a predetermined action in the user interface, such as selecting a menu option or clicking a logical button on the touch-sensitive display screen, or pressing a dedicated physical key on a keypad that toggles between e.g. numeric symbol set, upper-case letters and lower-case letters. The drawback with this alternative is that it requires the user to remove, the stylus from the current handwriting activity and move it to another location on the touch-sensitive display screen for selection of the menu option or clicking the logical button, or even dropping the stylus to depress the key on the keypad. Since the user's focus is momentarily switched from the handwriting activity to another action and then back again to handwriting, the quality of the handwriting will suffer both from a speed reduction and probably also a drop in accuracy.
An alternative is to write a special control stroke that has a predefined meaning and that will cause the handwriting functionality to interpret the succeeding handwriting input as belonging to a certain symbol set.
Still an alternative is to provide a control panel or mode selection bar, having selectable elements for the different available symbol sets. The control panel has a certain location preferably within or adjacent to the handwriting input area. By tapping with the stylus on a particular element in the control panel, the mode, i.e. symbol set, associated therewith will become the currently active one. This is intuitive to the user, since the meaning of each selectable element in the control panel can be visually indicated by designing the element as a graphical icon or button. Moreover, the control panel can conveniently be redesigned to represent new modes (symbol sets) by adding new elements to the control panel, or changing the meaning and visual appearance of existing ones. A drawback, however, is that the control panel will occupy a considerable part of the handwriting input area and will therefore reduce the actual available area for handwriting input.
U.S. Pat. No. 6,567,549, in
In view of the above, an objective of the invention is to solve or at least reduce the problems discussed above. More specifically, a purpose of the invention is to provide an improved manner for the user to control the current mode of handwriting in the apparatus, particularly so that the mode selection actions are intuitively and conveniently accessible to the user, preserving valuable space on the display screen at the same time.
Generally, the above objectives and purposes are achieved by an apparatus and a method for handwriting recognition according to the attached independent patent claims.
A first aspect of the invention is an apparatus for handwriting recognition, the apparatus comprising:
a display screen, such as a touch-sensitive display screen, providing a handwriting input area capable of detecting input from a user;
a processing device coupled to the display screen and providing a handwriting user interface to said user, said handwriting user interface being operable in at least a first mode and a second mode; and
a control panel in said handwriting user interface for selecting said first or second mode;
wherein said processing device is adapted to:
receive an input from said user,
detect, in the input received from said user, a control panel invoking command, in response to detecting the control panel invoking command, cause presentation of said control panel on said display screen, and
in response to a predetermined event, remove said control panel from presentation on said display screen.
In this way, whenever a switch in mode is desired, the user may conveniently perform the control panel invoking command to bring about the control panel. Then, once the predetermined event has occurred, the control panel will automatically disappear from the display screen. Thus, the control panel will only be present and occupy space on the display screen when it is needed, i.e. when the user is about to make a switch in mode. During other periods of time, the control panel will be absent and therefore not steal any valuable space on the display screen.
Advantageously, the control panel invoking command is a predefined handwriting action made by the user with a writing tool on the display screen. More specifically, in one embodiment the user may cause the predefined handwriting action by pointing with the writing tool at an arbitrary position on the display screen and keep the writing tool stationary at this position for at least a predetermined time period without removing it from pointing—i.e. making a “long-press” on the display screen with the writing tool. In a practical implementation, a “stationary” pointing position may be defined as a very small region confined to the immediate vicinity of the actual position at which the writing tool is first placed, to allow accidental minor movements of the writing tool caused by hand wobbling of a user which tries and intends to keep the writing tool still so as to command invocation of the control panel. In another embodiment the user may point a first time with his writing tool at an arbitrary position on the display screen, remove the writing tool from pointing at this position, and point a second time with the writing tool at substantially the same position within a predetermined time period—i.e. “double-clicking” on the display screen with the writing tool.
Alternatively, the predefined handwriting action may be caused by the user by performing a predetermined gesture with the writing tool on the display screen. Such a predetermined gesture may involve writing a predetermined symbol which is different from symbols for which handwriting recognition is performed—i.e. a control symbol. The predetermined gesture may otherwise involve writing a predetermined symbol having a size which is substantially different from a typical size of symbols for which handwriting recognition is performed, i.e. a symbol which is either much smaller or much larger than the typical size of ordinary symbols. Still an alternative is that the predetermined gesture involves drawing a stroke which crosses at least a predetermined part of said handwriting input area, such as a diagonal stroke across a major part of the handwriting input area.
In some embodiments, the predetermined gesture is configurable by the user. In other words, the user may himself decide the particulars of the or each gesture that will cause presentation of the control panel. To this end, the user may enter a special settings routine in the user interface and either input the desired gesture by writing its symbol on the display screen, or select the desired gesture from a group of predefined gestures/symbols. Making the predetermined gesture configurable by the user is beneficial in that it allows each user to use the gesture that is the most convenient to him. In turn, this is likely to increase both the input speed and the recognition accuracy of handwriting, since a convenient gesture often would mean a gesture that could be entered both rapidly and accurately by the particular user.
Advantageously, the predetermined event is the detection of an action made with a writing tool within said control panel or said handwriting input area. More specifically, in one embodiment, the control panel comprises a first selectable item for selection of said first mode and a second selectable item for selection of said second mode, wherein aforesaid action made with the writing tool is when the user selects one of the first and second items with the writing tool.
In an alternative embodiment, the predetermined event is the absence of an action made with a writing tool within said control panel during a predetermined time. In other words, if the user remains inactive until the lapse of a timeout period (the duration of which may be configurable by the user), this will trigger the removal of the control panel from the display screen.
The handwriting recognition preferably involves interpreting hand-written user input in the handwriting input area as a symbol among predefined symbols, wherein said first and second modes are associated with first and second sets of predefined symbols, respectively, to be used for the interpretation of hand-written user input.
These first and second sets of predefined symbols may be selected from the group consisting of: Latin characters, upper case characters, lower case characters, Arabic numerals, punctuation symbols, Cyrillic characters, Chinese symbols, Japanese Kanji symbols, Japanese Hiragana characters, Japanese Katakana characters, Korean Hangeul symbols, and user-defined symbols.
In some embodiments, the handwriting user interface is further operable in at least a third mode, said control panel being adapted for selecting between any of said first, second and third modes, wherein said third mode is associated with a third set of predefined symbols to be used for the interpretation of hand-written user input.
The control panel may have an adaptive location within said handwriting input area on said display screen. To this end, the processing device may be configured to adjust the adaptive location depending on at least one of the following: a current cursor position or a current point of actuation on said display screen with a writing tool. Alternatively, the control panel may have a fixed location within-the handwriting input area.
For maximum writing space, the handwriting input area is advantageously formed by a majority of the display screen's available presentation area, or even essentially the entire presentation area. In some embodiments, though, the handwriting input area may be limited to at least one dedicated handwriting character input box which occupies only a part of the display screen's entire available presentation area.
As used herein, “handwriting” means making a stroke, or a sequence of successive strokes within short time intervals, on the display screen by way of a writing tool in the form of a pen, stylus or any pen-like object including a user's finger or other body part. Such strokes are referred to as “pen strokes” in the remainder of this document.
The processing device may be configured to display, on the display screen, a graphical trace representing a pen stroke prior to the interpretation thereof. Moreover, the processing device may be configured to display, on the display screen, the symbol when it has been interpreted from the pen stroke.
The processing device advantageously includes or cooperates with a handwriting recognition engine which may be implemented as hardware, software or any combination thereof.
The apparatus may be a mobile terminal for a mobile telecommunications system, such as GSM, UMTS, D-AMPS or CDMA2000, or a portable/personal digital assistant (PDA), or another type of similar device.
A second aspect of the invention is a method for handwriting recognition in an apparatus having a display screen with a handwriting input area capable of detecting input from a user, the display screen being included in a handwriting user interface which is operable in at least a first mode and a second mode, the method involving the steps of:
receiving an input from said user;
detecting, in the input received from said user, a control panel invoking command;
causing presentation on said display screen of a control panel suitable for selection of said first or second mode;
detecting the occurrence of a predetermined event; and
removing said control panel from presentation on said display screen.
The second aspect has generally the same features and advantages as the first aspect.
Thus, the step of detecting the control panel invoking command may involve detecting a predefined handwriting action made with a writing tool on said display screen, e.g. detecting that said user points with said writing tool on said display screen at a stationary position thereon for at least a predetermined time period without removing said writing tool from pointing, or detecting that said user points a first time with said writing tool on said display screen at a position thereon, removes said writing tool from pointing at said position, and points a second time with said writing tool at said position within a predetermined time period.
Moreover, the step of detecting the control panel invoking command may involve detecting that said user performs a predetermined gesture with said writing tool on said display screen. Such a predetermined gesture may include one of the following: writing a predetermined symbol different from symbols for which handwriting recognition is performed; writing a predetermined symbol having a size which is substantially different from a typical size of symbols for which handwriting recognition is performed; and drawing a stroke which crosses at least a predetermined part of said handwriting input area.
The step of detecting the occurrence of a predetermined event may involve detecting an action made with a writing tool within said control panel or said handwriting input area. Alternatively, it may involve detecting that an action has not been made with a writing tool within said control panel during a predetermined time.
The control panel may comprise a first selectable item for selection of said first mode and a second selectable item for selection of said second mode, wherein the action made with said writing tool within said control panel involves selecting one of said first and second items.
When the handwriting recognition involves interpreting hand-written user input in the handwriting input area as a symbol among predefined symbols, the first and second modes may be associated with first and second sets of predefined symbols, respectively, to be used for the interpretation of hand-written user input. The handwriting user interface may further be operable in at least a third mode, wherein the control panel will be adapted for selecting between any of said first, second and third modes and wherein said third mode will be associated with a third set of predefined symbols to be used for the interpretation of hand-written user input.
The method may involve the steps of determining a current cursor position on said display screen; and adjusting an adaptive location of said control panel within said handwriting input area on said display screen depending on the determined cursor position. Alternatively, it may involve the steps of determining a current point of actuation on said display screen with a writing tool; and adjusting an adaptive location of said control panel within said handwriting input area on said display screen depending on the determined point of actuation.
Other objectives, features and advantages of the present invention will appear from the following detailed disclosure, from the attached dependent claims as well as from the drawings.
The present invention will now be described in more detail, reference being made to the enclosed drawings.
a is a schematic front view of an embodiment of the apparatus for handwriting recognition shown in
b-f are a schematic step-wise illustration of how handwriting is performed and how a mode-selecting control panel is invoked and used for selection between different symbol sets.
a-4i disclose a sequence of display screen snapshots taken from a practical implementation of the apparatus when used for inputting a hand-written text made of symbols from both a Chinese symbol set and a Latin character set.
a-5j disclose a sequence of display screen snapshots taken from another practical implementation of the apparatus when used for inputting the hand-written text of
A telecommunications system in which the present invention may be applied will first be described with reference to
In the telecommunications system of
In the illustrated example, the apparatus 100 may be used for speech communication with users of other devices. Hence, speech may be communicated with a user of a stationary telephone 132 through a public switched telephone network (PSTN) 130 and the mobile telecommunications network 110, and with a user of another mobile terminal 100′ which is connected to the mobile telecommunications network 110 over a wireless communication link 102′ to a base station 104′.
The mobile telecommunications network 110 is operatively connected to a wide area network 120, which may be Internet or a part thereof. Thus, the apparatus 100 may access a computer 122 connected to the wide area network 120 in accordance with specified protocols (such as TCP, IP and HTTP) and appropriate application software (such as a WAP or WWW browser, an email or SMS application, etc) in the apparatus 100.
The system illustrated in
a illustrates the apparatus 100 of
In more particular, by pointing, tapping, clicking or dragging a stylus 250 on the display screen 240, the user may use the stylus 250 as a logical mouse to control the user interface of the apparatus 100 by e.g. scrolling and selecting in different menus 260, 262, 264 and their menu options, setting the position of a cursor 256 on the display screen 240, actuating selectable control elements such as icons or click buttons 266, 268, selecting check boxes, controlling scroll bars, etc.
Moreover, the stylus 250 may be used as a logical pen to enter hand-written information within a handwriting input area 270. In the embodiment of
The hand-written information may be entered into various software applications, such as a messaging application (email, SMS, MMS), a calendar application, a notes or word processor application, a contacts application, etc. Hand-written input within the handwriting input area 270 is processed by a handwriting recognition engine in an attempt to interpret the input as a symbol out of a current symbol set. In the situation in
The user interface of the handwriting apparatus 100 is operable in different modes, each being associated with a respective symbol set to be used by the handwriting recognition engine when matching a handwritten input. The novel and inventive way in which the user selects mode, and thereby also the current symbol set, by way of dynamic provision of a mode-selecting control panel will now be explained with reference to
As seen in
However, moving back in time to the state shown in
Once the control panel invoking command has been detected (step 620), a control panel 280 will be shown on the display screen 240 (step 630), as seen in
Now, the user may conveniently select the desired symbol set by tapping with the stylus 250 on the corresponding one of the sub areas 282-288, i.e. by applying a logical-mouse pen-down. In
Because of the selection of this new mode, the next hand-written input, at 252 in
One of the available modes/symbol sets is preferably used as a default mode/symbol set, i.e. the mode/symbol set used at start-up or when the user has not made any active selection. Different applications, and different scenarios or input fields in the same application, may have different default modes/symbol sets. For instance, an application that predominantly handles numeric input, such as a telephone dialer or a calculator, may use Arabic numerals as default, whereas a text handling application may be defaulted to a symbol set which is determined e.g. by a general language setting for the user interface.
The functionality described above for dynamic provision of a mode-selecting control panel, as well as the handwriting recognition engine, may be included in the set of MMI drivers 334 or may be provided as separate software executable by the controller 300. A large variety of existing handwriting recognition algorithms and products, software-based and/or hardware-based, may be used to implement the handwriting recognition engine, as is readily realized by the skilled person.
The software also includes various modules, protocol stacks, drivers, etc., which are commonly designated as 330 and which provide communication services (such as transport, network and connectivity) for an RF interface 306, a Bluetooth interface 308 and an IrDA interface 310. The RF interface 306 comprises an internal or external antenna as well as appropriate radio circuitry for establishing and maintaining a wireless link to a base station (e.g. link 102 to base station 104 in
The apparatus 100 also has a SIM card 304 and an associated reader. As is commonly known, the SIM card 304 comprises a processor as well as local work and data memory.
The handwriting input referred to above may be received and used for various purposes in a variety of applications, including aforesaid messaging, calendar, notes and contacts applications 340, 342, 344 and 346, as well as for instance an Internet browser application, a WWW browser application, a WAP browser application, a phonebook application, a camera application, an imaging application, a video recording application, an organizer application, a video game application, a calculator application, a voice memo application, an alarm clock application, a word processing application, a code memory application, a music player application, a media streaming application, and a general control panel/settings application, or any other application which uses at least one field for text, character or symbol input.
The control panel 280 may be designed in many different ways. It may be divided into an arbitrary number of sub areas (2, 3, 4 (as in
The control panel 280 need not necessarily be designed as a horizontal bar but have other geometrical forms, for instance a vertical bar, a circle or a square box. If the control panel 280 is a square box, its sub areas may be positioned like quadrants in a coordinate system.
a-4i disclose a sequence of display screen snapshots taken from a practical implementation of the apparatus when used for inputting a hand-written text made of symbols from both a Chinese symbol set and a Latin character set into a notes application. Another practical implementation is shown in FIGS. a-5j. Like reference numerals represent the same or equivalent element in these FIGS as in
Then, in
The procedure is roughly the same in
The invention has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.
