Patent Application
20060271580
The present invention relates to the organization of freeform handwriting in electronic ink. Various aspects of the present invention are particularly applicable to organizing freeform electronic ink handwriting into lines.
Computers are regularly being used for a variety of purposes throughout the world. As computers have become commonplace, computer manufacturers have continuously sought to make them more accessible and user-friendly. One such effort has been the development of natural input methods, such as submitting data through handwriting. By writing with a stylus or another object onto a digitizer to produce “electronic ink,” a computer user can forego the bulk and inconvenience associated with a keyboard. Handwriting input conveniently may be used, for example, by doctors making rounds, architects on a building site, couriers delivering packages, warehouse workers walking around a warehouse, and in any situation when the use of a keyboard would be awkward or inconvenient. The use of handwriting input is particularly useful when the use of a keyboard and mouse would be inconvenient or inappropriate, such as when the writer is moving, in a quite meeting, or the like. The use of handwriting input also is the natural choice for creating some types of data, such as mathematical formulas, charts, drawings, and annotations.
While handwriting input is more convenient than keyboard input in many situations, text written in electronic ink typically cannot be directly manipulated by most software applications. Instead, text written in electronic ink must be analyzed to convert it into another form, such as ASCII characters. This analysis includes a handwriting recognition process, which recognizes characters based upon various relationships between individual electronic ink strokes making up a word of electronic ink. Handwriting recognition algorithms have improved dramatically in recent years, but their accuracy can be reduced when electronic ink is written at an angle. Likewise, when separate groups of ink strokes cannot be easily distinguished, such as when two words are written closely together, many recognition algorithms cannot accurately recognize electronic ink. Some recognition algorithms also may incorrectly recognize electronic ink as text when, in fact, the electronic ink is intended to be a drawing.
The accuracy of many recognition algorithms can be greatly improved by “parsing” (e.g., by analyzing the layout of and/or “classifying”) the electronic ink before using the handwriting recognition algorithm. A classification process typically determines whether an electronic ink stroke is part of a drawing (that is, a drawing ink stroke) or part of handwritten text (that is, a text ink stroke). Classification algorithms for identifying other stroke types also are possible. The layout analysis process typically groups electronic ink strokes into meaningful associations, such as words, lines and paragraphs.
Text lines are the most salient structures in freeform handwriting, and their reliable detection is the foundation to higher level layout analysis and semantic parsing. Freeform ink notes are a mixture of complex structures such as blocks of text, drawings, charts and annotations, and the combination of different structures often makes it difficult to reliably identify discrete lines of text in freeform handwriting. For example,
The handwriting 101 may be incorrectly organized by a existing handwriting parsing technique. For example, as seen in this figure, bullets 103A-103D have erroneously been organized into a single vertical line. Also, the group of text 105A has been organized into three separate lines 109A-109C. Similarly, the group of text 105D has been organized into three separate lines 111A-111C. This erroneous recognition of the handwriting organization may make it particularly difficult for a handwriting recognition engine to correctly recognize, for example, the bullets 103A-103D.
In addition to handwriting recognition, parsing functions can be used to select handwriting text for editing and other manipulation. Accordingly, the erroneous organization of the handwriting may cause the incorrect handwriting to be selected and manipulated in a grouping that is inconvenient or even detrimental for a user.
Advantageously, various aspects of the invention relate to techniques for efficiently and accurately organizing freeform handwriting into lines. Various examples of the invention employ global cost function designed to find the simplest partitioning of electronic ink stroke into line groups that also maximize the “goodness” of the resulting lines and the consistency of their configuration. Some embodiments of the invention measure the “goodness” of a line based upon its linear regression error and the horizontal and vertical compactness of the strokes making up the line. The line consistency configuration for a grouping of strokes is measured by the angle difference between neighboring groups. With various examples of the invention, the global cost function also takes into account the complexity of the stroke partitioning, measured by the number of lines into which the strokes are grouped.
An initial grouping of strokes is made, and the cost for this initial grouping is determined. With some examples of the invention, the strokes may initially be grouped using the one-dimensional array of strokes created based upon the temporal order in which the strokes were written. Various partitions of this array are created and the particular partition used for the initial grouping can be identified using a simplified version of the cost function by dynamic programming.
Alternate groupings of the initial stroke grouping are then generated. These grouping may be created, for example, by using grouping techniques that attempt to correct under and over grouping errors that may be included in the initial stroke grouping. The global cost of each of these alternate stroke groupings is then calculated, and the stroke grouping that produces the largest global cost decrease from the global cost of the original grouping is selected. The alternate grouping creation, cost determination and evaluation, and grouping selection process then is repeated until the global cost for new grouping alternates no longer decreases.
1B illustrate an example of handwriting that has been organized into stroke groups to form lines.
Overview
Various examples of the invention provide techniques for grouping strokes of electronic ink into lines. These techniques formulate line grouping as an optimal stroke partitioning problem. Thus, given a page of N strokes, the techniques determine the partitioning π* of the stroke set S among all possible partitions {π|π εII} to optimize the goodness of the resulting lines and the consistency and simplicity of their configuration.
Various notations will be used herein to describe different aspects of the invention. These notations include:
s: a stroke, s ε S, represented by an array of sample points pi=(xi, yi), i=1, 2, . . .Ns.
{l: a hypothesized line group, made up of a set of strokes {s: s ε Sl}.
π: a hypothesized line partitioning into groups {π|π ε II}, made up of a set of lines
{l:l ε π} where ∀(l,n):l,nεπ,l≠nSl∩Sn=Ø and ∪lεπ Sl=S.
Implementation Environment
As will be appreciated by those of ordinary skill in the art, various examples of the invention may be implemented using analog circuitry. More commonly, however, embodiments of the invention will be implemented using a programmable computing device executing software instructions. Accordingly, one example of this type of implementation environment for various embodiments of the invention will now be described with regard to
Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
As shown in
A basic input/output system (BIOS) 260 contains the basic routines that help to transfer information between elements within the computer 200, such as during start-up, is stored in the ROM 240. The computer 200 also may include a hard disk drive 270 for reading from and writing to a hard disk (not shown), a magnetic disk drive 280 for reading from or writing to a removable magnetic disk 290, and an optical disk drive 291 for reading from or writing to a removable optical disk 292, such as a CD ROM, DVD ROM, or other optical media. The hard disk drive 270, magnetic disk drive 280, and optical disk drive 291 are connected to the system bus 230 by a hard disk drive interface 292, a magnetic disk drive interface 293, and an optical disk drive interface 294, respectively. These drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules, and other data for the personal computer 200. It will be appreciated by those skilled in the art that other types of computer-readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the example operating environment.
A number of program modules can be stored on the hard disk drive 270, magnetic disk 290, optical disk 292, ROM 240, or RAM 250, including an operating system 295, one or more application programs 296, other program modules 297, and program data 298. A user can enter commands and information into the computer 200 through input devices, such as a keyboard 201 and pointing device 202 (such as a mouse). Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 210 through a serial port interface 206 that is coupled to the system bus 230, but they also may be connected by other interfaces, such as a parallel port, game port, or a universal serial bus (USB), and the like. Further still, these devices may be coupled directly to the system bus 230 via an appropriate interface (not shown).
A monitor 207 or other type of display device also may be connected to the system bus 230 via an interface, such as a video adapter 208. In addition to the monitor 207, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. In one example, a pen digitizer 265 and accompanying pen or stylus 266 are provided in order to digitally capture freehand input. Although a connection between the pen digitizer 265 and the serial port interface 206 is shown in
The computer 200 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 209. The remote computer 209 can be a server, a router, a network PC, a peer device or other common network node, and it typically includes many or all of the elements described above relative to the computer 200, although for simplicity, only a memory storage device 211 has been illustrated in
When used in a LAN networking environment, the computer 200 is connected to the local area network 212 through a network interface or adapter 214. When used in a WAN networking environment, the personal computer 200 typically includes a modem 215 or other means for establishing a communications link over the wide area network 213, such as the Internet. The modem 215, which may be internal or external to the computer 200, may be connected to the system bus 230 via the serial port interface 206. In a networked environment, program modules depicted relative to the personal computer 200, or portions thereof, may be stored in the remote memory storage device.
It will be appreciated that the network connections shown are examples and other techniques for establishing a communications link between the computers can be used. The existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP, UDP, and the like is presumed, and the system can be operated in a user-server configuration to permit a user to retrieve web pages from a web-based server. Any of various conventional web browsers can be used to display and manipulate data on web pages.
Although the
The stylus 266 may be equipped with buttons or other features to augment its capabilities. In one example, a stylus 266 could be implemented as a “pencil” or “pen”, in which one end constitutes a writing portion. The other end of the stylus 266 then constitutes an “eraser” end, which, when moved across the display, indicates portions of electronic ink on the display that are to be erased. Other types of input devices, such as a mouse, trackball, keyboard, or the like also could be used. Additionally, a user's own finger could be used for selecting or indicating portions of the displayed image if the display is a touch-sensitive or proximity-sensitive display. Consequently, the term “user input device,” as used herein, is intended to have a broad definition and encompasses many variations on well-known input devices.
In addition to use with full performance pen-based computing systems or “tablet PCs” (e.g., convertible laptops or “slate” type tablet PCs), aspects of this invention can be used in conjunction with other types of pen-based computing systems and/or other devices that accept data as electronic ink and/or accept electronic pen or stylus input, such as: hand-held or palm-top computing systems; personal digital assistants; pocket personal computers; mobile and cellular telephones, pagers, and other communication devices; watches; appliances; and any other devices or systems that include a monitor or other display device and/or a digitizer that presents printed or graphical information to users and/or allows input using an electronic pen or stylus, or which can process electronic ink collected by another device (e.g., a conventional desktop computer that can process electronic ink collected by a tablet PC).
Tool For Handwriting Recognition
The grouping cost determination module 405 then determines the cost of each stroke grouping created by the stroke grouping generation module 403. The grouping cost determination module 405 may calculate the cost for each stroke grouping using, for example, a global cost function, as will be explained in detail below. The grouping cost determination module 405 selects the alternate stroke grouping that provides the largest reduction in cost (if any), and provides the selected stroke grouping to the stroke grouping generation module 403. The stroke grouping generation module 403 then generates new alternate stroke groupings from the selected stroke grouping. The grouping cost determination module 405 then determines the cost for each of the new alternate groupings, and selects the new alternate stroke grouping that provides the largest reduction in cost (if any). The grouping cost determination module 405 then provides the new selected stroke grouping to the stroke grouping generation module 403. This process is repeated until the newest set of alternate stroke groupings does not reduce the cost from the currently selected stroke grouping. When this occurs, the currently selected stroke grouping is output as the stroke grouping selection 409.
Cost Determination
Referring now to
Various embodiments of the invention may use any desired cost function, but the illustrated embodiment employs a cost function that incorporates three “likelihood” terms. That is, these terms includes characteristics of a grouping of strokes prior characteristics reflecting the likelihood that the arrangement of groups is an accurate partitioning of the strokes into lines. The cost function also includes two “prior” terms, reflecting prior knowledge regarding line partitioning in freeform handwritten text. More particularly, the illustrated example of the cost determination module 405 employs a linear regression error and the maximum vertical and horizontal inter-stroke difference for each group of strokes as “likelihood” terms. It then employs line configuration consistency between the groups and the complexity of the grouping as “prior” terms.
For each group of strokes, the line regression error determination module 501 determines a linear regression error eLR by measuring the deviations of the points making up the strokes in the group from a fitting line. In the illustrated embodiment, for example, the line regression error determination module 501 uses the Total Least Squares technique to determine the goodness of a line. With the Total Least Squares linear regression technique, given the set of stroke points in a line l, the linear Total Least Squares linear regression technique finds the parameters (al, bl) to minimize the sum of the squared fitting error:
where eLR measure the deviations of the point from the fitting line and reflects the linearity of the stroke set. The line segment defined by the linear regression technique and the point boundaries as {circumflex over (l)}.
For example,
Using the fitting line 603, the maximum inter-stroke distances determination module 503 determines the maximum inter-stroke distances dxmax(l) and dymax(l). As shown in
For each group of strokes, the line configuration consistency determination module 505 determines the group's configuration consistency θ(l), which reflect the piecewise-smooth constraints of the stroke grouping. More particularly, the configuration consistency θ(l) of each group is calculated as the neighbor-length-weighted sum of the orientation angle differences between the group and each of neighbors, i.e.,
where α1 is the orientation angle of the line segment l determined from the fitting parameters (al, bl). |{circumflex over (l)}| is the length of the segment, often indicating the relative confidence of the line l. Thus, weights can be used to encourage a line group to be more consistent with it more reliable neighbors. In the above equation, ρ(r,σ) is a function regulating the influence of the residual r. In the illustrated example, ρ(r,σ=|r|.
The global consistency of a particular partitioning or arrangement of groups π can then be determined as the sum of all of the line consistencies as:
θπ=Σlεπθ(l)
For still other embodiments of the invention, more robust functions, such as, for example, the Geman-McClure function: rho(r, sigma)=r-squared/(r-squared+sigma-squared), can be adopted to regulate the angle difference to reduce oversmoothing, where neighboring line groups have very different orientation (e.g., such as might occur between main text and annotation text).
To measure the consistency of a line configuration, the line configuration consistency determination module 505 creates a neighborhood graph. In this graph, each vertex corresponds to a group of strokes forming a line. The edges of the graph then correspond to neighbor relationships between the line groups. With various examples of the invention, a pair of line groups are considered neighbors if the minimum distance between their fitting line segments 601 falls below a threshold and there are no other objects (lines or drawings) lying between them. Still other embodiments of the invention may determine groups of strokes are neighbors if they are direct neighbors (i.e., with no other objects in between), or within certain Euclidian distances, or combinations of two or more different criteria.
With various examples of the invention, the global consistency of a particular partitioning or arrangement of groups π can alternately be expressed as
where e denotes an edge in the neighborhood graph G, and le1, le2 denote the two nodes that e connects. Instead of summing over all nodes in the graph, this expression sums over all edges.
In the illustrated embodiment, the cost determination module 507 calculates a cost associated for a partitioning of ink into line groups using the weighted sum of the linear regression error eLR, the maximum inter-stroke distances dxmax(l) and dymax(l), and the configuration consistency θ (l) of each group, taking into account the complexity of the grouping. The complexity π of a handwritten ink is the number of line groups into which the ink has been partitioned. Thus, the illustrated example of the cost determination module 507 determines the cost of a stroke grouping as:
where wi are parameters of the formulation controlling the relative importance of the terms. With various examples of the invention, w1=1, w2=1, w3=10000, and w4=100, but these values may be changed according to training/tuning of the tool. As will be explained in more detail below, the optimal grouping of strokes is the one that minimizes the cost obtained using this function.
Generating Stroke Groupings
As noted above, the stroke grouping generation module 403 generates an initial or baseline grouping of strokes. With some examples of the invention, this initial grouping of strokes is obtained by partitioning the one dimensional array of temporally ordered strokes into alternate arrangements of groups to form lines. Each alternate group arrangement then can be analyzed using a cost function. The stroke grouping generation module 403 may, for example, use a cost function that employs a subset of the terms in the global cost function used by the cost determination module 507 described in detail above. For example, the stroke grouping generation module 403 may use the function
where dxmax(l) is the maximum inter stroke distance projected on to the fitting line 601. Such a simplification is enabled by the dramatic reduction of the search space, and it also gives the cost function a recursive structure so that dynamic programming can apply. As a practical speedup, the stroke sequence can be divided into spans based on stroke proximity and size similarity, and perform partitioning only within each span.
Once the initial stroke grouping has been determined, the stroke grouping generation module 403 then generates alternate groupings of the strokes based upon the original stroke grouping using, for example, gradient-descent approach. For example, various implementations of the invention may generate two types of alternative groupings. The first type of alternate grouping is obtained by merging pairs of line groups that are neighbors in the neighborhood graph, discussed above. This type of alternate grouping tends to correct under-grouping errors that are typical in temporal grouping results. Under grouping errors may be caused by, for example, late ‘i’ dots and ‘t’ crosses. The stroke grouping generation module 403 generates a merging hypothesis alternate stroke grouping for each neighboring pair in the initial stroke grouping.
The second type of alternate grouping is formed by attempting to correct any high configuration energy errors in the initial stroke grouping. High configuration energy errors are caused by temporally adjacent strokes belonging to different lines. These errors are frequently caused by, for example, bullets written before or after the list content is filled. Thus, any set of neighboring line groups where one of the groups (e.g., group “X”) has a maximum angle difference from its neighbors which exceeds a threshold value (e.g., 30°) is an initial candidate. That is, the stroke grouping generation module 403 identifies those neighbor groups of group X that are approximately parallel to each other, but which have substantial angle differences with the group X. The stroke grouping generation module 403 then uses these neighbor groups as bins to split the strokes in Group X.
Some examples of the invention may alternately or additionally create alternate groupings of a third type to separate alternate groupings of a third type to separate y-outliers. With these alternates, in a group of strokes forming a line segment l, with a nonzero dymax value, the strokes are split at the maximum y-gap. If the smaller stroke set l0 is much closer to a neighbor group n than to its current line group l, then the smaller stroke set l0 is merged with the neighbor group n.
Determining Cost Changes
As discussed above, each alternate stroke grouping changes an existing stroke grouping π0 to an alternative stroke grouping π1 by re-grouping strokes in a local neighborhood. As illustrated in
Given a selected stroke grouping π01 at iteration i (which may be the initial stroke grouping or a subsequently selected stroke grouping), the stroke grouping generation module 403 generates a queue of alternative stroke groupings {πji, j=1,2, . . .}. Next, the grouping cost determination module 405 identifies accept the stroke grouping π*i that provides the largest global cost reduction. The stroke grouping generation module 403 then creates a new set of alternate stroke groupings accordingly, and this process is iterated until the queue is empty.
More particularly, given a selected stroke grouping π0i at iteration i, the stroke grouping generation module 403 generates a queue of alternative stroke group arrangements {πji,j=1,2, . . .}. Next, the grouping cost determination module 405 identifies accept the stroke grouping π*i that provides the largest global cost reduction. If this stroke grouping π*i does not reduce the global cost relative to the global cost for π0i, then the process stops and the arrangement π0i is selected as the line partitioning. If not, then the arrangement π*i is accepted as the next iteration. The graph is updated by removing {l0} from the graph, together with all affected edges. The new line {l* } for the new alternate arrangement is created in the graph, together with the appropriate edges.
Next, the queue of alternate arrangements is then updated. This includes removing conflicting alternate arrangements. An arrangement of strokes into groups πji conflicts with π*i if lj0∩l*0≠Ø. Then, the affected alternate arrangement is updated. An arrangement πji is affected by π*i if its Naffectedj∩Ø. The configuration costs are then computed, and new alternate arrangements are generated. This becomes the next iteration, where i→i+1.
Once the arrangement of line groups having the lowest cost function is determined, then the data structure reflecting the relationships between the strokes of electronic ink is updated to reflect this line grouping.
Incremental Stroke Organization
While the examples of the invention described above relate to a “batch” processing of electronic ink, it should be appreciated that these local gradient-descent optimization techniques also provide a good solution for incremental parsing of electronic ink. This use may be particularly beneficial for real-time ink analysis. Online ink data is usually generated incrementally. Each time that an ink parser is called, the input data may be a page of parsed results plus several new strokes. It is desirable to have some incremental mechanism to “massage” the new strokes into the existing page structure. The relaxation-like optimization algorithm described above performs this task naturally. With some implementations of the invention, this incremental operation may be much more efficient than the batch operations with consistent results.
Conclusion
While the invention has been described with respect to specific examples including presently preferred modes of carrying out the invention, those skilled in the art will appreciate that there are numerous variations and permutations of the above described systems and techniques that fall within the spirit and scope of the invention as set forth in the appended claims.
