The present invention relates to a system for tracking and processing handwritten pen strokes on a mobile terminal.
Digital writing instruments (or styluses), interchangeably referred to herein as “digital pens” regardless of whether or not they write in ink, can be used to capture pen strokes and to digitize them. Digital writing systems including character recognition software can be used to convert recorded pen strokes to image or text data.
Typically, digital pens operate with and send data to a host terminal (such as a personal computer) through a wired connection, as a supplement to keyboard or mouse input. This may be directly or indirectly via one or more receiving stations which receive a signal from the digital pen. The location of the digital pen is tracked with a terminal tracking mechanism, and location data may be written into memory of the host terminal.
On mobile terminals having touch screen such as Smartphone's, PDA's and tablet PC's, it is possible to capture handwriting notes and sketches by touching the screen with a finger or with a stylus. The capture of handwriting directly on the touch screen presents several limitations: ergonomic issues, very low precision, screen much smaller than typical paper notepad, etc.
Many mobile terminals do not offer the possibility to receive pen strokes from a digital pen as they have no proper port for the wired connection. In the case of USB data transfer, even if the mobile computer and the digital pen have an USB port, neither can be configured as a host. In any case, the use of a wired connection between a mobile terminal and an external device is usually seen as a major limitation to the ease of use of it.
It is an aim of this invention to provide a system which does not show at least one of the above mentioned drawbacks.
According to the present invention, the aforementioned aim is achieved with a system for tracking and processing handwritten pen strokes, comprising; a digital pen to allow a user to write pen strokes on a substrate; a receiver station for use in combination with said digital pen and adapted for tracking said pen strokes and storing digital pen strokes which comprise electronic data representing said tracked pen strokes, said receiver station comprising a wireless transceiver adapted for communicating with a mobile terminal over a wireless connection to transfer said digital pen strokes to said mobile terminal; and a software application executable on said mobile terminal and comprising software code fragments for receiving and processing said transferred digital pen strokes.
In embodiments according to the present invention, the processing may comprise one or more of the following: image processing, character recognition, image compression, and file format conversion to a text format or any format combining image and text, including the processed data in an e-mail, storing for further processing or other. The further processing may comprise all of the above and can be performed on the mobile terminal (embedded processing) or externally, by using for example a SaaS (Software as a Service) delivery model.
In embodiments according to the present invention, the wireless connection between the receiver station and the mobile terminal can be a Bluetooth® connection, a Wifi connection or any other wireless connection. Preferably an RF wireless connection with a guaranteed bandwidth (or data throughput) is used, for smooth transfer of the pen strokes.
In embodiments according to the present invention, the digital pen strokes can contain position information and a time-stamp, so that the order of the pen strokes can be reconstructed.
The invention will be further elucidated by means of the following description and the appended drawings.
The present invention will be described with respect to particular embodiments and with reference to certain drawings but the invention is not limited thereto but only by the claims. The drawings described are only schematic and are non-limiting. In the drawings, the size of some of the elements may be exaggerated and not drawn on scale for illustrative purposes. The dimensions and the relative dimensions do not necessarily correspond to actual reductions to practice of the invention.
Furthermore, the terms first, second, third and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. The terms are interchangeable under appropriate circumstances and the embodiments of the invention can operate in other sequences than described or illustrated herein.
Moreover, the terms top, bottom, over, under and the like in the description and the claims are used for descriptive purposes and not necessarily for describing relative positions. The terms so used are interchangeable under appropriate circumstances and the embodiments of the invention described herein can operate in other orientations than described or illustrated herein.
Furthermore, the various embodiments, although referred to as “preferred” are to be construed as exemplary manners in which the disclosure may be implemented rather than as limiting the scope of the invention.
The term “comprising”, used in the claims, should not be interpreted as being restricted to the means listed thereafter; it does not exclude other elements or steps. It needs to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression “a device comprising means A and B” should not be limited to devices consisting only of components A and B. It means that with respect to the present invention, the only relevant components of the device are A and B.
The mobile terminal 4 is wirelessly connected with the receiver station 3 which is used in combination with the digital pen 1, and has software code fragments for capturing and recording the pen strokes that are written by the user on the substrate 2, and can display the resulting graphical image 10 on a display 5. The display is preferably integrated in the mobile terminal 4, e.g. the LCD screen of an ipad, but may also be an external device. The screen 5 may be a touch-screen or not. The resolution of the screen 5 is not related to the resolution (or accuracy) of the captured pen strokes. This means that the handwritten pen-strokes may be captured in much more detail and with a much higher accuracy than what is possible by using the touch-screen as an input device.
In an embodiment of the system 12, the mobile terminal 4 is provided with an application software for compressing the reconstructed graphical image 10 for archiving (e.g. on a hard disk or a flash card or other memory or an external network drive, etc) or for sharing the image 10, e.g. by sending it over a network 6, e.g. via e-mail.
In an embodiment of the system 12, the application software uses (optical) character recognition techniques in order to get the digital text data 11 corresponding with the handwritten notes 13. The digital text data 11 may e.g. be represented as ASCII-character codes (without any formatting information such as color, font-type, font-size, etc), which may be saved as a “text-only” file. Alternatively the digital text data 11 may also comprise formatting data, such as e.g. color or font-size or font-type, and may e.g. be saved as a rich-text format (e.g. an “RFT-file”), or as a Microsoft® Msword® document, or as other document formats, e.g. WordPerfect®, MsWorks®, or other. But the digital text data 11 may also be represented as unicode characters, which allows also handwriting 13 of language such as Chinese or Japanese to be further processed.
The character recognition software which is used may convert the graphical data (such as e.g. a bitmap 10 with a graphical representation of the handwritten notes 13) into text data 11 along with position information and not keep the accompanied drawings (graphical data 10). The latter may however contain important information that the writer has sketched. Since character recognition software can make conversion mistakes (e.g. a character mismatch), the digital image 10 of the handwriting notes 13 may be needed in order to keep a correct reference. So, it is preferred to use character recognition software which keeps both the converted text 11 and the (graphical) image 10 of the handwriting note together, for archiving or for sending by e-mail 18.
In an embodiment of the system, illustrated in
In an embodiment of the system, the image 10 of the handwritten notes 13 is combined with other images 7 such as e.g. the background image of a form to be filled out, or a picture (e.g. a photograph) or a screen-shot, or a scanned image, or any other images. An example is shown in
Therefore in an embodiment of the system of the present invention, the system software code also comprises code fragments for compressing the image 15 resulting from the combination of the graphical image 10 of the reconstructed pen strokes and the other images 7, by using a high compression method that segments the image 15 into bi-tonal (pen strokes, text, graphics) data and color (text and graphic colors, background color, pictures) data and compress those data separately with a compression method adapted to the data type. The high compression method could follow the MRC (Mixed Raster Content) model, object of the ITU-T recommendation T.44.
The receiver station 3 may be a battery-powered device with internal memory, (e.g. volatile memory such as DRAM, or non-volatile memory such as flash-memory), capable of working in stand-alone mode (i.e. without being connected to the mobile terminal 4) for temporarily storing tracked pen strokes 13. The pen strokes are recorded in the memory of the receiver station 3. The pen-strokes can later be uploaded to the mobile terminal 4, saved or emailed as ordinary files. This system 12 allows handwritten email composition. According to the invention the pen strokes are imported in the mobile terminal 4 using the wireless link 9. Thereto the software code fragments of the mobile terminal 4 incorporate the protocol stack of the wireless link technology being used.
One particularly interesting short-range wireless communications protocol is widely known as Bluetooth®. Bluetooth technology, also known as IEEE 802.15.1 operates in the unlicensed industrial, scientific and medical (ISM) band at 2.4 GHz, and uses a spread spectrum technique to minimize interference. The core specification (core specification v2.0+EDR, published 4 Nov. 2004) for Bluetooth is available at http://bluetooth.com. The content of this core specification is hereby incorporated by reference in its entirety. Another interesting wireless communications protocol is Wifi, also known as IEEE 802.11 a/b/g/n. Preferably the mobile terminal 4 has an embedded RF transceiver.
The receiver station and mobile terminal can communicate by using the Bluetooth protocol implemented at either side by the protocol stack that manages the timing critical radio interface and deals with the high level data.
The receiver station can comprise a radio which is arranged for communicating with the mobile terminal, suitably via the Bluetooth wireless protocol. The radio is connected to a microprocessor arranged for controlling the transmission of signals between the receiver station and the mobile terminal.
The receiver station can comprise a silicon device containing the bluetooth radio and a microprocessor that implements the protocol stack.
Mobile terminals that support Bluetooth comprise a radio device and a protocol stack. Their protocol stack is implemented as part of their operating system and offers an API (Application Programming Interface) to the application software. The application software uses the API to verify or request that Bluetooth is enabled by the user, setup Bluetooth, find the digital pen device, connect to the digital pen device and manage the connection.
When uploading the recorded pen stroke data, the wireless link 9 is preferably configured in such as way that the data is transferred at the highest speed in the shortest possible time, while ensuring a reliable transmission.
Preferably the receiver station 3 can also be configured in a connected mode, or in some embodiments may only have the connected mode, (i.e. connected to the mobile terminal 4). In this mode the pen strokes are continuously tracked and transferred as soon as possible to the mobile terminal 4 via the wireless link 9. In this mode, the wireless link 9 is preferably configured in such a way that the delay between the pen and the mobile terminal 4 is minimized. Alternatively the wireless link 9 may also be configured in a low-power mode, for saving battery lifetime of the mobile terminal and/or the battery lifetime of the receiver station 3.
In an embodiment, the pen 1 may e.g. be configured as a writing device, whereby a force or pressure sensor built inside the pen tip is used for detecting when the pen tip is actually being used for writing, (which pen strokes 13 are then recorded). Optionally the pen 1 may also be configured for use as a pointing device like a mouse, whereby all pen movements are transferred to the mobile terminal 4 in real time. Optionally the pen 1 may also have a push button to simulate a mouse click.
In an embodiment the receiver station 3 stores a sequence of digitized positions of handwritten pen strokes 13 in the memory, and groups them as “virtual pages” corresponding to physical papers being written upon. Preferably the receiver station 3 has means for changing the “virtual page”, e.g. using a push-button, such that when the button is pushed, a new virtual page is started. The receiver station 3 may comprise sufficient memory to store e.g. 50 or 100 or more virtual pages, which may be ideal for students for taking notes in a classroom, while having the ability to edit them later.
The mobile terminal 4 may operate on a standard Microsoft® Windows® operation system, such as Microsoft® Windows XP®, or Windows 7®, but other operating systems may also be used, such as e.g. iOS, Android, Blackberry OS, Windows Phone 7, HP's webOS, or other.
The above mentioned software application code fragments may be stand-alone programs, or may be incorporated or embedded in driver software, or may be provided as plug-ins for cooperating with existing software applications such as e.g. Powerpoint® which is part of Office2007® from Microsoft®, or :Evernote® from Evernote Corporation but may also be provided in other ways known to the person skilled in the art.
Number | Date | Country | Kind |
---|---|---|---|
2011/0638 | Oct 2011 | BE | national |
Number | Date | Country | |
---|---|---|---|
61490671 | May 2011 | US |