Patent Application
20190155895
Pen-enabled devices are becoming more commonplace and display digitizers are becoming standardized components of laptop computers and other computing devices. Pen input allows handwriting of text, which can often be more conducive to creative endeavors and brainstorming than using a traditional keyboard and mouse. However, various opportunities remain for creating an electronic handwriting input environment that moves beyond the realm of pen and paper by harnessing the speed and power of modern computing devices.
Aspects of the subject technology provide systems and methods for providing a text pen mode providing unobtrusive writing guidance for improved legibility and organization. In the text pen mode, a guideline may be shown that corresponds to a detected writing baseline, which may adjust over time as additional writing is provided. By default, the guideline may be horizontally straight, but the text pen mode may also support angled, curved, or irregular guidelines. A goal of the text pen mode is to encourage creativity and freeform mixed layouts where drawings, figures, and text may be placed anywhere in the document. To that end, the guideline may extend out a certain distance according to various considerations, including heuristics, machine learning algorithms, user writing style, surrounding context, etc., rather than extending all the way out to the page margin.
Aspects of the subject technology provide a computer-implemented method that includes receiving, from an input component of an electronic device, a pen input. The method also includes determining that an input mode for the pen input is a text-input mode. The method also includes identifying a feature of at least one character in the pen input. The method also includes providing for display, on a display device of the electronic device, a representation of the pen input and a guideline. The guideline extends along and adjacent to at least a portion of the representation of the pen input and beyond an end of the representation of the pen input by a length that is based on the identified feature of the at least one character.
Aspects of the subject technology provide a system. The system includes an input component, a display device, a memory comprising instructions, and a processor. The processor is configured to execute the instructions to receive, from the input component, a pen input. The processor is further configured to execute the instructions to identify a size and a writing rate of at least one character in the pen input. The processor is further configured to execute the instructions to provide for display, on the display device, a representation of the pen input and a guideline. The guideline extends along and adjacent to at least a portion of the representation of the pen input and beyond an end of the representation of the pen input by a length that is based on the identified size and writing rate of the at least one character
Aspects of the subject technology provide a non-transitory machine-readable medium, which when executed by one or more processors, cause the processors to perform operations that include receiving, from an input component of an electronic device, a pen input. The operations also include identifying features of handwritten text in the pen input. The operations also include providing for display, on a display device of the electronic device, a representation of the handwritten text and a first guideline. The first guideline extends along and adjacent to at least a portion of the representation of the handwritten text. The operations also include providing a second guideline, separated from the first guideline by a distance that is based on the identified features of the handwritten text. The second guideline has a length that is based on the identified features of the handwritten text.
Further aspects of the subject technology, features, and advantages, as well as the structure and operation of various aspects of the subject technology are described in detail below with reference to accompanying drawings.
Aspects of the subject technology are described with reference to the accompanying drawings. In the drawings, like reference numbers may indicate identical or functionally similar elements.
While aspects of the subject technology are described herein with reference to illustrative examples for particular applications, it should be understood that the subject technology is not limited to those particular applications. Those skilled in the art with access to the teachings provided herein will recognize additional modifications, applications, and aspects within the scope thereof and additional fields in which the subject technology would be of significant utility.
Pen-enabled document editing applications may present a user with a blank canvas or a rigidly organized document template similar to lined ruled paper. However, with a blank canvas, it may be difficult for the user to devise any meaningful structure, resulting in a messy or disorganized document that may cause the user to perform revisions and/or corrections that can be frustratingly time consuming and can waste computing power. On the other hand, with a rigid template such as an electronic template that mimics lined ruled paper, the user may be discouraged from drawing figures, diagrams, or other creative content that deviates from the template's guidance.
The subject disclosure provides systems and methods for providing a text pen mode in a pen-enabled editor application that extracts information from current or previously provided pen inputs to provide real-time adaptive guidance for inputting handwritten text with a text pen device.
Computing device 101 includes processor 121, one or more input/output components such as display device 122, communications module 123, and memory 130. Memory 130 includes pen-enabled editor application 140, which may include guideline features package 142.
In aspects of the subject technology, computing device 101 is a pen-enabled device that may receive pen input from pen device 178 via display device 122. For example, computing device 101 may be a desktop all-in-one computer, tablet or convertible computer, laptop computer, smart phone, portable gaming console, or other device having a display device 122 supporting pen input.
In this description, pen input is used to refer generally to input from pen device 178, which may be implemented as a pen or a stylus that can interface with display device 122 of computing device 101. For example, pen device 178 may be an active device that includes processing circuitry, memory, and/or one or more applications stored in the pen device memory, or may be a passive device having a material composition configured to be detectable by circuitry within display device 122. For example, a passive pen device may interface with capacitive or resistive touchscreens, display digitizers, peripheral tablet devices, or other input devices coupled to or integrated with display device 122. In some aspects of the subject technology, display device 122 may correspond to a display projector or augmented reality eyewear, in which case pen device 178 be implemented as an air pen that provides pen input without contacting a physical surface.
A user desiring to hand write text with pen device 178 and computing device 101 may, for example, access an application storefront supported by application server 180 to download and install pen-enabled editor application 140 onto computing device 101. Thus, computing device 101 may utilize communications module 123, for example an Ethernet adapter, wireless card or cellular modem, to communicate with application server 180 over network 170. In aspects of the subject technology, pen-enabled editor application 140 may be provided on installable media, for example on an optical disc, flash memory card, or other device. In other aspects of the subject technology, pen-enabled editor application 140 may be preinstalled onto computing device 101, or pen-enabled editor application 140 may be integrated as part of an operating system of computing device 101.
As indicated in
As shown in
Guideline features package 142 includes features for pen-enabled editor application 140 that help provide unobtrusive writing guidance to a user entering handwritten text, for improved legibility and organization of a document. In pen-enabled editor application 140, a “text pen” mode may be selectable by a user or may be or automatically started when the application detects writing of text (e.g., by detecting at least one character in a pen input from pen device 178). A character may include a letter, a number, or a symbol corresponding to any of various human languages.
Once in the text pen mode, pen-enabled editor application 140 provides, at least when text input is being provided to the application using pen device 178, a guideline that is generated responsive to the entering of the text input. The guideline may be provided at a location, and having an adaptive length and/or shape that is dependent on one or more features of the entered text. The guideline extends beneath (e.g., along and adjacent to a bottom portion of) at least a portion of a representation of text provided in a pen input and extends beyond an end of the representation of the text by a length that is based one or more features of one or more handwritten characters. By default, the guideline may be horizontally straight (e.g., parallel to a top and/or bottom edge of a virtual page provided by the application), but the text pen mode may also, or alternatively, provide angled, curved, or irregular guidelines depending on the arrangement of the entered text. The guideline may grow, shrink, and/or change shapes in real time with the entry of the handwritten text to adapt to the direction, shape, speed, and size (and/or other features) of the text as it is being written.
The guideline may extend through one or more baseline locations associated with one or more entered characters. For example, a baseline location may be separated vertically from a detected bottom of an “a” or a “k” or may pass through the stem of a “y” (as examples).
Providing an adaptive, real-time guideline may help encourage creativity and freeform mixed layouts where drawings, figures, and text may be placed anywhere in the document, while harnessing the power of a modern computing device to improve (e.g., the neatness and/or organization) the ultimate digital representation of the handwritten input. To that end, the guideline may extend out a certain distance according to various considerations, including heuristics, machine learning algorithms, user writing style, surrounding context, etc., rather than extending all the way out to the page margin of a virtual page of application 140.
Extending portion 126 is provided as a guide for the user to enter additional text, associated with the already entered text “The quick” at a location and in a direction that is consistent with the location and direction of the already entered text and encourages neatness and legibility in the ultimate digital document.
As shown, extending portion 126 includes a fading portion 128 that fades in darkness, texture, and/or color into a background on which the text is written, with increasing distance from the last character. As shown, extending portion 126 has a length L. L may be an adaptive length that grows, shrinks, appears, disappears, changes shape, and/or otherwise evolves based on the text that is written by the user. Guideline 120 may appear when a first character in a pen input from pen device 178 is detected. For example, pen device 178 can be used to draw pictures or other non-character entries that do not cause a guideline to be displayed by pen-enabled editor application 140.
For example, length L may be based on the width of one or more of the characters of the already entered text “The quick” (e.g., a median or mean length of the characters or a width of the last character). Length L may also be based on a speed or rate at which the characters are being written. For example, extending portion 126 of guideline 120 may have a length that is equal to a multiplicative factor times the determined character width. The multiplicative factor may be determined based on the writing rate. For example, extending portion 126 may extend ten character widths beyond the last written character for a moderate writing rate, twenty characters beyond the last written character for a fast writing rate, or five characters beyond the last written character for a slow writing rate.
Length L can be modified with the entry of each written character or after each set of N written characters (e.g., the length can be modified after each set of three written characters, five written characters, or ten written characters). As shown, guideline 120 has an overall length (including extended portion 126) that is less than a maximum line length LMAX. Maximum line length LMAX may be, for example a page width (or a width inside a drawable area of a page as bounded by automatic or user-defined margins) of a virtual page provided by application 140.
Portion 124 of guideline 120 also grows in length as additional characters are written.
Guideline 120b may appear earlier during the writing on guideline 120a for faster writers than for slower writers. Guideline 120b of
Although not displayed in
As shown in
Text box 500 may be generated in the background by application 140, but not displayed to the user, unless the user provides an additional input (e.g., a finger tap or other non-writing input) that causes application 140 to display the text box. When the text box is displayed, the text box may be manipulable so that the handwritten text in the received pen input and/or the associated guidelines can be encapsulated to be selected, dragged, dropped, rotated, resized, or otherwise manipulated by the user by moving, rotating, resizing or otherwise manipulating the text box.
In the examples of
In the example of
In the example of
As indicated in
An angled and/or curved guideline 120 includes a portion 124 that runs along and adjacent to the text and an extending portion 126 that extends from portion 124 and beyond an end of the text (i.e., beyond the character “d” in the example of
Pen-enabled editor application 140 may generate guidelines that avoid extending into other document elements, such as other text boxes or drawings that have previously been placed or entered.
Additional guideline 120b may discourage the user from writing over element 1100. However, as shown in
When application 140 determines that the user has stopped entering text, application 140 may redisplay the text that was entered over element 1100 on the guidelines 120a and 120b as shown in
In the example of
Although examples have been described in which an additional guideline is provided when a current guideline reaches a maximum length, in some circumstances application 140 may provide only a single guideline. For example,
As shown in
In the example of
Examples have been described in which the boundaries to which a guideline provided by application 140 can extend are defined by the edges of a virtual page or the edge of another document element. However, guidelines can also be generated responsive to user-generated boundaries.
For example,
In the example of
Examples have been described in which the boundaries to which a guideline provided by application 140 can extend are defined by the edges of a virtual page, the edge of another document element, or a user-generated boundary. However, guidelines can also be generated based on a vertical distribution of previously-entered text.
For example,
As the user begins writing “Pros” (e.g., with pen device 178), guideline 1704 may be generated as described above in connection with the “The quick” example of
The user may also enter text for the sub-columns in each primary column. In the example of
Sub-column guidelines 1708 and 1710 each have an individual length that is determined based on the width of the sub-column entries. For example, guideline 1708 may include a portion that extends along and adjacent to the text “Great” and an extending portion that extends beyond the last “character” to a maximum length (e.g., a maximum length equal to an average of the widths of the previous column entries or a width of a maximum-length entry in the sub-column). Based on the writing rate and completeness of the previous column entries, guideline 1710 may be provided (e.g., when guideline 1708 approaches or reaches the maximum length).
In the example of
Sub-column guidelines 1716 and 1718 each have an individual length that is determined based on the width of the sub-column entries as described in connection with guidelines 1708 and 1710. For example, guideline 1716 may include a portion that extends along and adjacent to the text “Strange” and an extending portion that extends beyond the last character “e” to a maximum length (e.g., a maximum length equal to an average of the widths of the previous column entries or a width of a maximum-length entry in the sub-column). Based on the writing rate and completeness of the previous column entries, guideline 1718 may be provided (e.g., when guideline 1716 approaches or reaches the maximum length).
As shown in
Although, in the example described above in connection with
It should also be appreciated that, although the examples described above in connection with
In the depicted example flow diagram, at block 1811, a pen input such as a pen input from pen device 178 of
At block 1812, the pen-enabled editor application detects at least one character in the pen input. The pen input may include other input such as a drawing, a boundary, or other hand-drawable input. The at least one character may include one or more letters, numbers, symbols, or other characters such as one or more of the characters of the text “The quick” as described in various examples here (see, e.g.,
At block 1813, the pen-enabled editor application enters a text input mode, responsive to detecting the at least one character. It should also be appreciated that the text input mode may be selectable by the user (e.g., before any characters of text are provided in a pen input). In the text input mode, additional pen input is interpreted as handwritten text and various operations for providing guidance to the user for additional text input are performed by the application.
For example, at block 1814, the pen-enabled editor application detects one or more aspects of the at least one character. For example, the pen-enabled editor application may detect aspects such as the height and/or width of each character, the size of the spaces between one or more pairs of adjacent characters, an angle with respect to a horizontal or vertical edge of the virtual page, a curved direction of entry of multiple characters, and/or a writing rate (e.g., a rate in letters per second or words per minute) of the at least one character.
At block 1815, the pen-enabled editor application provides, for display on a display device (e.g., display device 122), a representation of the pen input (see, e.g., representation 110 as described herein in connection with various examples) and a guideline (see, e.g., guidelines 120, 120a, 120b, 1704, 1706, 1708, 1710, 1712, 1716, and 1718 as described herein in connection with various examples). The guideline extends beyond an end of the displayed representation of the pen input by a length that is based on the detected aspect(s) of the pen input as described above in connection with, for example,
Bus 1908 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of electronic system 1900. For instance, bus 1908 communicatively connects processing unit(s) 1912 with ROM 1910, system memory 1904, and permanent storage device 1902.
From these various memory units, processing unit(s) 1912 retrieves instructions to execute and data to process in order to execute the processes of the subject disclosure. The processing unit(s) can be a single processor or a multi-core processor in different implementations.
ROM 1910 stores static data and instructions that are needed by processing unit(s) 1912 and other modules of the electronic system. Permanent storage device 1902, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when electronic system 1900 is off. Some implementations of the subject disclosure use a mass-storage device (for example, a magnetic or optical disk and its corresponding disk drive) as permanent storage device 1902.
Other implementations use a removable storage device (for example, a floppy disk, flash drive, and its corresponding disk drive) as permanent storage device 1902. Like permanent storage device 1902, system memory 1904 is a read-and-write memory device. However, unlike storage device 1902, system memory 1904 is a volatile read-and-write memory, such a random access memory. System memory 1904 stores some of the instructions and data that the processor needs at runtime. In some implementations, the processes of the subject disclosure are stored in system memory 1904, permanent storage device 1902, or ROM 1910. For example, the various memory units include instructions for processing a social endorsement for an item in accordance with some implementations. From these various memory units, processing unit(s) 1912 retrieves instructions to execute and data to process in order to execute the processes of some implementations.
Bus 1908 also connects to input and output device interfaces 1914 and 1906. Input device interface 1914 enables the user to communicate information and select commands to the electronic system. Input devices used with input device interface 1914 include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”). Output device interfaces 1906 enables, for example, the display of images generated by the electronic system 1900. Output devices used with output device interface 1906 include, for example, printers and display devices, for example, cathode ray tubes (CRT) or liquid crystal displays (LCD). Some implementations include devices, for example, a touchscreen that functions as both input and output devices.
Finally, as shown in
Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.
In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some implementations, multiple software aspects of the subject disclosure can be implemented as sub-parts of a larger program while remaining distinct software aspects of the subject disclosure. In some implementations, multiple software aspects can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software aspect described here is within the scope of the subject disclosure. In some implementations, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.
A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
These functions described above can be implemented in digital electronic circuitry, in computer software, firmware or hardware. The techniques can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. The processes and logic flows can be performed by one or more programmable processors and by one or more programmable logic circuitry. General and special purpose computing devices and storage devices can be interconnected through communication networks.
Some implementations include electronic components, for example, microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media can store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, for example, is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some implementations are performed by one or more integrated circuits, for example, application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification and any claims of this application, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this application, the terms “computer readable medium” and “computer readable media” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.
To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.
Aspects of the subject technology described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In aspects of the subject technology, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.
It is understood that any specific order or hierarchy of steps in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged, or that all illustrated steps be performed. Some of the steps may be performed simultaneously. For example, in certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in aspects of the subject technology described above should not be understood as requiring such separation in all aspects of the subject technology, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. Pronouns in the masculine (e.g., his) include the feminine and neuter gender (e.g., her and its) and vice versa. Headings and subheadings, if any, are used for convenience only and do not limit the subject disclosure.
A phrase such as an “aspect” does not imply that such aspect is essential to the subject technology or that such aspect applies to all configurations of the subject technology. A disclosure relating to an aspect may apply to all configurations, or one or more configurations. A phrase such as an aspect may refer to one or more aspects and vice versa. A phrase such as a “configuration” does not imply that such configuration is essential to the subject technology or that such configuration applies to all configurations of the subject technology. A disclosure relating to a configuration may apply to all configurations, or one or more configurations. A phrase such as a configuration may refer to one or more configurations and vice versa.
