Patent Grant
12093523
This relates generally to electronic devices that accept handwritten inputs, and user interactions with such devices.
User interaction with electronic devices has increased significantly in recent years. These devices can be devices such as computers, tablet computers, televisions, multimedia devices, mobile devices, and the like.
In some circumstances, users wish to input text on an electronic device or otherwise interact with an electronic device with a stylus. In some circumstances, users wish to use a stylus or other handwriting device to handwrite desired text onto the touch screen display of the electronic device. Enhancing these interactions improves the user's experience with the device and decreases user interaction time, which is particularly important where input devices are battery-operated.
It is well understood that the use of personally identifiable information should follow privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of users. In particular, personally identifiable information data should be managed and handled so as to minimize risks of unintentional or unauthorized access or use, and the nature of authorized use should be clearly indicated to users.
Some embodiments described in this disclosure are directed to receiving handwritten inputs in text entry fields and converting the handwritten inputs into font-based text. Some embodiments described in this disclosure are directed to selecting and deleting text using a stylus. Some embodiments of the disclosure are directed to inserting text into pre-existing text using a stylus. Some embodiments of the disclosure are directed to managing the timing of converting handwritten inputs into font-based text. Some embodiments of the disclosure are directed to presenting, on an electronic device, a handwritten entry menu. Some embodiments of the disclosure are directed to controlling the characteristic of handwritten inputs based on selections on the handwritten entry menu. Some embodiments of the disclosure are directed to presenting autocomplete suggestions. Some embodiments of the disclosure are directed to converting handwritten input to font-based text. Some embodiments of the disclosure are directed to displaying options in a content entry palette.
For a better understanding of the various described embodiments, reference should be made to the Detailed Description below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures.
The following description sets forth exemplary methods, parameters, and the like. It should be recognized, however, that such description is not intended as a limitation on the scope of the present disclosure but is instead provided as a description of exemplary embodiments.
There is a need for electronic devices that provide efficient methods for receiving and interpreting handwritten inputs (e.g., from a stylus or other handwriting input device). Such techniques can reduce the cognitive burden on a user who uses such devices. Further, such techniques can reduce processor and battery power otherwise wasted on redundant user inputs.
Although the following description uses terms “first,” “second,” etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first touch could be termed a second touch, and, similarly, a second touch could be termed a first touch, without departing from the scope of the various described embodiments. The first touch and the second touch are both touches, but they are not the same touch.
The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The term “if” is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event],” depending on the context.
Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as PDA and/or music player functions. Exemplary embodiments of portable multifunction devices include, without limitation, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and/or touchpads), are, optionally, used. It should also be understood that, in some embodiments, the device is not a portable communications device, but is a desktop computer with a touch-sensitive surface (e.g., a touch screen display and/or a touchpad).
In the discussion that follows, an electronic device that includes a display and a touch-sensitive surface is described. It should be understood, however, that the electronic device optionally includes one or more other physical user-interface devices, such as a physical keyboard, a mouse, and/or a joystick.
The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.
The various applications that are executed on the device optionally use at least one common physical user-interface device, such as the touch-sensitive surface. One or more functions of the touch-sensitive surface as well as corresponding information displayed on the device are, optionally, adjusted and/or varied from one application to the next and/or within a respective application. In this way, a common physical architecture (such as the touch-sensitive surface) of the device optionally supports the variety of applications with user interfaces that are intuitive and transparent to the user.
Attention is now directed toward embodiments of portable devices with touch-sensitive displays.
As used in the specification and claims, the term “intensity” of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a substitute (proxy) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values that includes at least four distinct values and more typically includes hundreds of distinct values (e.g., at least 256). Intensity of a contact is, optionally, determined (or measured) using various approaches and various sensors or combinations of sensors. For example, one or more force sensors underneath or adjacent to the touch-sensitive surface are, optionally, used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., a weighted average) to determine an estimated force of a contact. Similarly, a pressure-sensitive tip of a stylus is, optionally, used to determine a pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and/or changes thereto, the capacitance of the touch-sensitive surface proximate to the contact and/or changes thereto, and/or the resistance of the touch-sensitive surface proximate to the contact and/or changes thereto are, optionally, used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the substitute measurements for contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurements). In some implementations, the substitute measurements for contact force or pressure are converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of a contact as an attribute of a user input allows for user access to additional device functionality that may otherwise not be accessible by the user on a reduced-size device with limited real estate for displaying affordances (e.g., on a touch-sensitive display) and/or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical/mechanical control such as a knob or a button).
As used in the specification and claims, the term “tactile output” refers to physical displacement of a device relative to a previous position of the device, physical displacement of a component (e.g., a touch-sensitive surface) of a device relative to another component (e.g., housing) of the device, or displacement of the component relative to a center of mass of the device that will be detected by a user with the user's sense of touch. For example, in situations where the device or the component of the device is in contact with a surface of a user that is sensitive to touch (e.g., a finger, palm, or other part of a user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in physical characteristics of the device or the component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is, optionally, interpreted by the user as a “down click” or “up click” of a physical actuator button. In some cases, a user will feel a tactile sensation such as an “down click” or “up click” even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movements. As another example, movement of the touch-sensitive surface is, optionally, interpreted or sensed by the user as “roughness” of the touch-sensitive surface, even when there is no change in smoothness of the touch-sensitive surface. While such interpretations of touch by a user will be subject to the individualized sensory perceptions of the user, there are many sensory perceptions of touch that are common to a large majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., an “up click,” a “down click,” “roughness”), unless otherwise stated, the generated tactile output corresponds to physical displacement of the device or a component thereof that will generate the described sensory perception for a typical (or average) user.
It should be appreciated that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. The various components shown in
Memory 102 optionally includes high-speed random access memory and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.
Peripherals interface 118 can be used to couple input and output peripherals of the device to CPU 120 and memory 102. The one or more processors 120 run or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data. In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are, optionally, implemented on a single chip, such as chip 104. In some other embodiments, they are, optionally, implemented on separate chips.
RF (radio frequency) circuitry 108 receives and sends RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to/from electromagnetic signals and communicates with communications networks and other communications devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and so forth. RF circuitry 108 optionally communicates with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication. The RF circuitry 108 optionally includes well-known circuitry for detecting near field communication (NFC) fields, such as by a short-range communication radio. The wireless communication optionally uses any of a plurality of communications standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and/or IEEE 802.11ac), voice over Internet Protocol (VoIP), Wi-MAX, a protocol for e-mail (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.
Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100. Audio circuitry 110 receives audio data from peripherals interface 118, converts the audio data to an electrical signal, and transmits the electrical signal to speaker 111. Speaker 111 converts the electrical signal to human-audible sound waves. Audio circuitry 110 also receives electrical signals converted by microphone 113 from sound waves. Audio circuitry 110 converts the electrical signal to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data is, optionally, retrieved from and/or transmitted to memory 102 and/or RF circuitry 108 by peripherals interface 118. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., 212,
I/O subsystem 106 couples input/output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripherals interface 118. I/O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive/send electrical signals from/to other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate embodiments, input controller(s) 160 are, optionally, coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. The one or more buttons (e.g., 208,
A quick press of the push button optionally disengages a lock of touch screen 112 or optionally begins a process that uses gestures on the touch screen to unlock the device, as described in U.S. patent application Ser. No. 11/322,549, “Unlocking a Device by Performing Gestures on an Unlock Image,” filed Dec. 23, 2005, U.S. Pat. No. 7,657,849, which is hereby incorporated by reference in its entirety. A longer press of the push button (e.g., 206) optionally turns power to device 100 on or off. The functionality of one or more of the buttons are, optionally, user-customizable. Touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.
Touch-sensitive display 112 provides an input interface and an output interface between the device and a user. Display controller 156 receives and/or sends electrical signals from/to touch screen 112. Touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively termed “graphics”). In some embodiments, some or all of the visual output optionally corresponds to user-interface objects.
Touch screen 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on haptic and/or tactile contact. Touch screen 112 and display controller 156 (along with any associated modules and/or sets of instructions in memory 102) detect contact (and any movement or breaking of the contact) on touch screen 112 and convert the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages, or images) that are displayed on touch screen 112. In an exemplary embodiment, a point of contact between touch screen 112 and the user corresponds to a finger of the user.
Touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. Touch screen 112 and display controller 156 optionally detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 112. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, California.
A touch-sensitive display in some embodiments of touch screen 112 is, optionally, analogous to the multi-touch sensitive touchpads described in the following U.S. Pat. No. 6,323,846 (Westerman et al.), U.S. Pat. No. 6,570,557 (Westerman et al.), and/or U.S. Pat. No. 6,677,932 (Westerman), and/or U.S. Patent Publication 2002/0015024A1, each of which is hereby incorporated by reference in its entirety. However, touch screen 112 displays visual output from device 100, whereas touch-sensitive touchpads do not provide visual output.
A touch-sensitive display in some embodiments of touch screen 112 is described in the following applications: (1) U.S. patent application Ser. No. 11/381,313, “Multipoint Touch Surface Controller,” filed May 2, 2006; (2) U.S. patent application Ser. No. 10/840,862, “Multipoint Touchscreen,” filed May 6, 2004; (3) U.S. patent application Ser. No. 10/903,964, “Gestures For Touch Sensitive Input Devices,” filed Jul. 30, 2004; (4) U.S. patent application Ser. No. 11/048,264, “Gestures For Touch Sensitive Input Devices,” filed Jan. 31, 2005; (5) U.S. patent application Ser. No. 11/038,590, “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices,” filed Jan. 18, 2005; (6) U.S. patent application Ser. No. 11/228,758, “Virtual Input Device Placement On A Touch Screen User Interface,” filed Sep. 16, 2005; (7) U.S. patent application Ser. No. 11/228,700, “Operation Of A Computer With A Touch Screen Interface,” filed Sep. 16, 2005; (8) U.S. patent application Ser. No. 11/228,737, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard,” filed Sep. 16, 2005; and (9) U.S. patent application Ser. No. 11/367,749, “Multi-Functional Hand-Held Device,” filed Mar. 3, 2006. All of these applications are incorporated by reference herein in their entirety.
Touch screen 112 optionally has a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. The user optionally makes contact with touch screen 112 using any suitable object or appendage, such as a stylus, a finger, and so forth. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen. In some embodiments, the device translates the rough finger-based input into a precise pointer/cursor position or command for performing the actions desired by the user.
In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad (not shown) for activating or deactivating particular functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output. The touchpad is, optionally, a touch-sensitive surface that is separate from touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.
Device 100 also includes power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)) and any other components associated with the generation, management and distribution of power in portable devices.
Device 100 optionally also includes one or more optical sensors 164.
Device 100 optionally also includes one or more contact intensity sensors 165.
Device 100 optionally also includes one or more proximity sensors 166.
Device 100 optionally also includes one or more tactile output generators 167.
Device 100 optionally also includes one or more accelerometers 168.
In some embodiments, the software components stored in memory 102 include operating system 126, communication module (or set of instructions) 128, contact/motion module (or set of instructions) 130, graphics module (or set of instructions) 132, text input module (or set of instructions) 134, Global Positioning System (GPS) module (or set of instructions) 135, and applications (or sets of instructions) 136. Furthermore, in some embodiments, memory 102 (
Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.
Communication module 128 facilitates communication with other devices over one or more external ports 124 and also includes various software components for handling data received by RF circuitry 108 and/or external port 124. External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted for coupling directly to other devices or indirectly over a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to and/or compatible with, the 30-pin connector used on iPod® (trademark of Apple Inc.) devices.
Contact/motion module 130 optionally detects contact with touch screen 112 (in conjunction with display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). Contact/motion module 130 includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining an intensity of the contact (e.g., the force or pressure of the contact or a substitute for the force or pressure of the contact), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact). Contact/motion module 130 receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, optionally includes determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations are, optionally, applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., “multitouch”/multiple finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detect contact on a touchpad.
In some embodiments, contact/motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has “clicked” on an icon). In some embodiments, at least a subset of the intensity thresholds are determined in accordance with software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of particular physical actuators and can be adjusted without changing the physical hardware of device 100). For example, a mouse “click” threshold of a trackpad or touch screen display can be set to any of a large range of predefined threshold values without changing the trackpad or touch screen display hardware. Additionally, in some implementations, a user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and/or by adjusting a plurality of intensity thresholds at once with a system-level click “intensity” parameter).
Contact/motion module 130 optionally detects a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and/or intensities of detected contacts). Thus, a gesture is, optionally, detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event followed by detecting a finger-up (liftoff) event at the same position (or substantially the same position) as the finger-down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and subsequently followed by detecting a finger-up (liftoff) event.
Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual property) of graphics that are displayed. As used herein, the term “graphics” includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations, and the like.
In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is, optionally, assigned a corresponding code. Graphics module 132 receives, from applications etc., one or more codes specifying graphics to be displayed along with, if necessary, coordinate data and other graphic property data, and then generates screen image data to output to display controller 156.
Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator(s) 167 to produce tactile outputs at one or more locations on device 100 in response to user interactions with device 100.
Text input module 134, which is, optionally, a component of graphics module 132, provides soft keyboards for entering text in various applications (e.g., contacts 137, e-mail 140, IM 141, browser 147, and any other application that needs text input).
GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to telephone 138 for use in location-based dialing; to camera 143 as picture/video metadata; and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map/navigation widgets).
Applications 136 optionally include the following modules (or sets of instructions), or a subset or superset thereof:
Examples of other applications 136 that are, optionally, stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.
In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, contacts module 137 are, optionally, used to manage an address book or contact list (e.g., stored in application internal state 192 of contacts module 137 in memory 102 or memory 370), including: adding name(s) to the address book; deleting name(s) from the address book; associating telephone number(s), e-mail address(es), physical address(es) or other information with a name; associating an image with a name; categorizing and sorting names; providing telephone numbers or e-mail addresses to initiate and/or facilitate communications by telephone 138, video conference module 139, e-mail 140, or IM 141; and so forth.
In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, telephone module 138 are optionally, used to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in contacts module 137, modify a telephone number that has been entered, dial a respective telephone number, conduct a conversation, and disconnect or hang up when the conversation is completed. As noted above, the wireless communication optionally uses any of a plurality of communications standards, protocols, and technologies.
In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact/motion module 130, graphics module 132, text input module 134, contacts module 137, and telephone module 138, video conference module 139 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions.
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, e-mail client module 140 includes executable instructions to create, send, receive, and manage e-mail in response to user instructions. In conjunction with image management module 144, e-mail client module 140 makes it very easy to create and send e-mails with still or video images taken with camera module 143.
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, the instant messaging module 141 includes executable instructions to enter a sequence of characters corresponding to an instant message, to modify previously entered characters, to transmit a respective instant message (for example, using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephony-based instant messages or using XMPP, SIMPLE, or IMPS for Internet-based instant messages), to receive instant messages, and to view received instant messages. In some embodiments, transmitted and/or received instant messages optionally include graphics, photos, audio files, video files and/or other attachments as are supported in an MMS and/or an Enhanced Messaging Service (EMS). As used herein, “instant messaging” refers to both telephony-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module, workout support module 142 includes executable instructions to create workouts (e.g., with time, distance, and/or calorie burning goals); communicate with workout sensors (sports devices); receive workout sensor data; calibrate sensors used to monitor a workout; select and play music for a workout; and display, store, and transmit workout data.
In conjunction with touch screen 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact/motion module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions to capture still images or video (including a video stream) and store them into memory 102, modify characteristics of a still image or video, or delete a still image or video from memory 102.
In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images.
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions to browse the Internet in accordance with user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, e-mail client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to-do lists, etc.) in accordance with user instructions.
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, and browser module 147, widget modules 149 are mini-applications that are, optionally, downloaded and used by a user (e.g., weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or created by the user (e.g., user-created widget 149-6). In some embodiments, a widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets).
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, and browser module 147, the widget creator module 150 are, optionally, used by a user to create widgets (e.g., turning a user-specified portion of a web page into a widget).
In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions to search for text, music, sound, image, video, and/or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.
In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present, or otherwise play back videos (e.g., on touch screen 112 or on an external, connected display via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).
In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, and text input module 134, notes module 153 includes executable instructions to create and manage notes, to-do lists, and the like in accordance with user instructions.
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact/motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 are, optionally, used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data on stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.
In conjunction with touch screen 112, display controller 156, contact/motion module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, e-mail client module 140, and browser module 147, online video module 155 includes instructions that allow the user to access, browse, receive (e.g., by streaming and/or download), play back (e.g., on the touch screen or on an external, connected display via external port 124), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 141, rather than e-mail client module 140, is used to send a link to a particular online video. Additional description of the online video application can be found in U.S. Provisional Patent Application No. 60/936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed Jun. 20, 2007, and U.S. patent application Ser. No. 11/968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed Dec. 31, 2007, the contents of which are hereby incorporated by reference in their entirety.
Each of the above-identified modules and applications corresponds to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules are, optionally, combined or otherwise rearranged in various embodiments. For example, video player module is, optionally, combined with music player module into a single module (e.g., video and music player module 152,
In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad. By using a touch screen and/or a touchpad as the primary input control device for operation of device 100, the number of physical input control devices (such as push buttons, dials, and the like) on device 100 is, optionally, reduced.
The predefined set of functions that are performed exclusively through a touch screen and/or a touchpad optionally include navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates device 100 to a main, home, or root menu from any user interface that is displayed on device 100. In such embodiments, a “menu button” is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.
Event sorter 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which to deliver the event information. Event sorter 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates the current application view(s) displayed on touch-sensitive display 112 when the application is active or executing. In some embodiments, device/global internal state 157 is used by event sorter 170 to determine which application(s) is (are) currently active, and application internal state 192 is used by event sorter 170 to determine application views 191 to which to deliver event information.
In some embodiments, application internal state 192 includes additional information, such as one or more of: resume information to be used when application 136-1 resumes execution, user interface state information that indicates information being displayed or that is ready for display by application 136-1, a state queue for enabling the user to go back to a prior state or view of application 136-1, and a redo/undo queue of previous actions taken by the user.
Event monitor 171 receives event information from peripherals interface 118. Event information includes information about a sub-event (e.g., a user touch on touch-sensitive display 112, as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I/O subsystem 106 or a sensor, such as proximity sensor 166, accelerometer(s) 168, and/or microphone 113 (through audio circuitry 110). Information that peripherals interface 118 receives from I/O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.
In some embodiments, event monitor 171 sends requests to the peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and/or for more than a predetermined duration).
In some embodiments, event sorter 170 also includes a hit view determination module 172 and/or an active event recognizer determination module 173.
Hit view determination module 172 provides software procedures for determining where a sub-event has taken place within one or more views when touch-sensitive display 112 displays more than one view. Views are made up of controls and other elements that a user can see on the display.
Another aspect of the user interface associated with an application is a set of views, sometimes herein called application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of a respective application) in which a touch is detected optionally correspond to programmatic levels within a programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is, optionally, called the hit view, and the set of events that are recognized as proper inputs are, optionally, determined based, at least in part, on the hit view of the initial touch that begins a touch-based gesture.
Hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies a hit view as the lowest view in the hierarchy which should handle the sub-event. In most circumstances, the hit view is the lowest level view in which an initiating sub-event occurs (e.g., the first sub-event in the sequence of sub-events that form an event or potential event). Once the hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.
Active event recognizer determination module 173 determines which view or views within a view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of a sub-event are actively involved views, and therefore determines that all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if touch sub-events were entirely confined to the area associated with one particular view, views higher in the hierarchy would still remain as actively involved views.
Event dispatcher module 174 dispatches the event information to an event recognizer (e.g., event recognizer 180). In embodiments including active event recognizer determination module 173, event dispatcher module 174 delivers the event information to an event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores in an event queue the event information, which is retrieved by a respective event receiver 182.
In some embodiments, operating system 126 includes event sorter 170. Alternatively, application 136-1 includes event sorter 170. In yet other embodiments, event sorter 170 is a stand-alone module, or a part of another module stored in memory 102, such as contact/motion module 130.
In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a respective view of the application's user interface. Each application view 191 of the application 136-1 includes one or more event recognizers 180. Typically, a respective application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a user interface kit (not shown) or a higher level object from which application 136-1 inherits methods and other properties. In some embodiments, a respective event handler 190 includes one or more of: data updater 176, object updater 177, GUI updater 178, and/or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views 191 include one or more respective event handlers 190. Also, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in a respective application view 191.
A respective event recognizer 180 receives event information (e.g., event data 179) from event sorter 170 and identifies an event from the event information. Event recognizer 180 includes event receiver 182 and event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of: metadata 183, and event delivery instructions 188 (which optionally include sub-event delivery instructions).
Event receiver 182 receives event information from event sorter 170. The event information includes information about a sub-event, for example, a touch or a touch movement. Depending on the sub-event, the event information also includes additional information, such as location of the sub-event. When the sub-event concerns motion of a touch, the event information optionally also includes speed and direction of the sub-event. In some embodiments, events include rotation of the device from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation (also called device attitude) of the device.
Event comparator 184 compares the event information to predefined event or sub-event definitions and, based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definitions 186. Event definitions 186 contain definitions of events (e.g., predefined sequences of sub-events), for example, event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in an event (187) include, for example, touch begin, touch end, touch movement, touch cancellation, and multiple touching. In one example, the definition for event 1 (187-1) is a double tap on a displayed object. The double tap, for example, comprises a first touch (touch begin) on the displayed object for a predetermined phase, a first liftoff (touch end) for a predetermined phase, a second touch (touch begin) on the displayed object for a predetermined phase, and a second liftoff (touch end) for a predetermined phase. In another example, the definition for event 2 (187-2) is a dragging on a displayed object. The dragging, for example, comprises a touch (or contact) on the displayed object for a predetermined phase, a movement of the touch across touch-sensitive display 112, and liftoff of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.
In some embodiments, event definition 187 includes a definition of an event for a respective user-interface object. In some embodiments, event comparator 184 performs a hit test to determine which user-interface object is associated with a sub-event. For example, in an application view in which three user-interface objects are displayed on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user-interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object triggering the hit test.
In some embodiments, the definition for a respective event (187) also includes delayed actions that delay delivery of the event information until after it has been determined whether the sequence of sub-events does or does not correspond to the event recognizer's event type.
When a respective event recognizer 180 determines that the series of sub-events do not match any of the events in event definitions 186, the respective event recognizer 180 enters an event impossible, event failed, or event ended state, after which it disregards subsequent sub-events of the touch-based gesture. In this situation, other event recognizers, if any, that remain active for the hit view continue to track and process sub-events of an ongoing touch-based gesture.
In some embodiments, a respective event recognizer 180 includes metadata 183 with configurable properties, flags, and/or lists that indicate how the event delivery system should perform sub-event delivery to actively involved event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate how event recognizers interact, or are enabled to interact, with one another. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate whether sub-events are delivered to varying levels in the view or programmatic hierarchy.
In some embodiments, a respective event recognizer 180 activates event handler 190 associated with an event when one or more particular sub-events of an event are recognized. In some embodiments, a respective event recognizer 180 delivers event information associated with the event to event handler 190. Activating an event handler 190 is distinct from sending (and deferred sending) sub-events to a respective hit view. In some embodiments, event recognizer 180 throws a flag associated with the recognized event, and event handler 190 associated with the flag catches the flag and performs a predefined process.
In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver event information to event handlers associated with the series of sub-events or to actively involved views. Event handlers associated with the series of sub-events or with actively involved views receive the event information and perform a predetermined process.
In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates the telephone number used in contacts module 137, or stores a video file used in video player module. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates a new user-interface object or updates the position of a user-interface object. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends it to graphics module 132 for display on a touch-sensitive display.
In some embodiments, event handler(s) 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of a respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.
It shall be understood that the foregoing discussion regarding event handling of user touches on touch-sensitive displays also applies to other forms of user inputs to operate multifunction devices 100 with input devices, not all of which are initiated on touch screens. For example, mouse movement and mouse button presses, optionally coordinated with single or multiple keyboard presses or holds; contact movements such as taps, drags, scrolls, etc. on touchpads; pen stylus inputs; movement of the device; oral instructions; detected eye movements; biometric inputs; and/or any combination thereof are optionally utilized as inputs corresponding to sub-events which define an event to be recognized.
In some embodiments, stylus 203 is an active device and includes one or more electronic circuitry. For example, stylus 203 includes one or more sensors, and one or more communication circuitry (such as communication module 128 and/or RF circuitry 108). In some embodiments, stylus 203 includes one or more processors and power systems (e.g., similar to power system 162). In some embodiments, stylus 203 includes an accelerometer (such as accelerometer 168), magnetometer, and/or gyroscope that is able to determine the position, angle, location, and/or other physical characteristics of stylus 203 (e.g., such as whether the stylus is placed down, angled toward or away from a device, and/or near or far from a device). In some embodiments, stylus 203 is in communication with an electronic device (e.g., via communication circuitry, over a wireless communication protocol such as Bluetooth) and transmits sensor data to the electronic device. In some embodiments, stylus 203 is able to determine (e.g., via the accelerometer or other sensors) whether the user is holding the device. In some embodiments, stylus 203 can accept tap inputs (e.g., single tap or double tap) on stylus 203 (e.g., received by the accelerometer or other sensors) from the user and interpret the input as a command or request to perform a function or change to a different input mode.
Device 100 optionally also include one or more physical buttons, such as “home” or menu button 204. As described previously, menu button 204 is, optionally, used to navigate to any application 136 in a set of applications that are, optionally, executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touch screen 112.
In some embodiments, device 100 includes touch screen 112, menu button 204, push button 206 for powering the device on/off and locking the device, volume adjustment button(s) 208, subscriber identity module (SIM) card slot 210, headset jack 212, and docking/charging external port 124. Push button 206 is, optionally, used to turn the power on/off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlock process. In an alternative embodiment, device 100 also accepts verbal input for activation or deactivation of some functions through microphone 113. Device 100 also, optionally, includes one or more contact intensity sensors 165 for detecting intensity of contacts on touch screen 112 and/or one or more tactile output generators 167 for generating tactile outputs for a user of device 100.
Each of the above-identified elements in
Attention is now directed towards embodiments of user interfaces that are, optionally, implemented on, for example, portable multifunction device 100.
It should be noted that the icon labels illustrated in
Although some of the examples that follow will be given with reference to inputs on touch screen display 112 (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface that is separate from the display, as shown in
Additionally, while the following examples are given primarily with reference to finger inputs (e.g., finger contacts, finger tap gestures, finger swipe gestures), it should be understood that, in some embodiments, one or more of the finger inputs are replaced with input from another input device (e.g., a mouse-based input or stylus input). For example, a swipe gesture is, optionally, replaced with a mouse click (e.g., instead of a contact) followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is, optionally, replaced with a mouse click while the cursor is located over the location of the tap gesture (e.g., instead of detection of the contact followed by ceasing to detect the contact). Similarly, when multiple user inputs are simultaneously detected, it should be understood that multiple computer mice are, optionally, used simultaneously, or a mouse and finger contacts are, optionally, used simultaneously.
Exemplary techniques for detecting and processing touch intensity are found, for example, in related applications: International Patent Application Serial No. PCT/US2013/040061, titled “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an application,” filed May 8, 2013, published as WIPO Publication No. WO/2013/169849, and International Patent Application Serial No. PCT/US2013/069483, titled “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed Nov. 11, 2013, published as WIPO Publication No. WO/2014/105276, each of which is hereby incorporated by reference in their entirety.
In some embodiments, device 500 has one or more input mechanisms 506 and 508. Input mechanisms 506 and 508, if included, can be physical. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, can permit attachment of device 500 with, for example, hats, eyewear, earrings, necklaces, shirts, jackets, bracelets, watch straps, chains, trousers, belts, shoes, purses, backpacks, and so forth. These attachment mechanisms permit device 500 to be worn by a user.
Input mechanism 508 is, optionally, a microphone, in some examples. Personal electronic device 500 optionally includes various sensors, such as GPS sensor 532, accelerometer 534, directional sensor 540 (e.g., compass), gyroscope 536, motion sensor 538, and/or a combination thereof, all of which can be operatively connected to I/O section 514.
Memory 518 of personal electronic device 500 can include one or more non-transitory computer-readable storage mediums, for storing computer-executable instructions, which, when executed by one or more computer processors 516, for example, can cause the computer processors to perform the techniques described below, including processes 700, 900, 1100, 1300, 1500, 1600, 1800, 2000, and 2200 (
As used here, the term “affordance” refers to a user-interactive graphical user interface object that is, optionally, displayed on the display screen of devices 100, 300, and/or 500 (
As used herein, the term “focus selector” refers to an input element that indicates a current part of a user interface with which a user is interacting. In some implementations that include a cursor or other location marker, the cursor acts as a “focus selector” so that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in
As used in the specification and claims, the term “characteristic intensity” of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is, optionally, based on a predefined number of intensity samples, or a set of intensity samples collected during a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) relative to a predefined event (e.g., after detecting the contact, prior to detecting liftoff of the contact, before or after detecting a start of movement of the contact, prior to detecting an end of the contact, before or after detecting an increase in intensity of the contact, and/or before or after detecting a decrease in intensity of the contact). A characteristic intensity of a contact is, optionally, based on one or more of: a maximum value of the intensities of the contact, a mean value of the intensities of the contact, an average value of the intensities of the contact, a top 10 percentile value of the intensities of the contact, a value at the half maximum of the intensities of the contact, a value at the 90 percent maximum of the intensities of the contact, or the like. In some embodiments, the duration of the contact is used in determining the characteristic intensity (e.g., when the characteristic intensity is an average of the intensity of the contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an operation has been performed by a user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, a contact with a characteristic intensity that does not exceed the first threshold results in a first operation, a contact with a characteristic intensity that exceeds the first intensity threshold and does not exceed the second intensity threshold results in a second operation, and a contact with a characteristic intensity that exceeds the second threshold results in a third operation. In some embodiments, a comparison between the characteristic intensity and one or more thresholds is used to determine whether or not to perform one or more operations (e.g., whether to perform a respective operation or forgo performing the respective operation), rather than being used to determine whether to perform a first operation or a second operation.
In some embodiments, a portion of a gesture is identified for purposes of determining a characteristic intensity. For example, a touch-sensitive surface optionally receives a continuous swipe contact transitioning from a start location and reaching an end location, at which point the intensity of the contact increases. In this example, the characteristic intensity of the contact at the end location is, optionally, based on only a portion of the continuous swipe contact, and not the entire swipe contact (e.g., only the portion of the swipe contact at the end location). In some embodiments, a smoothing algorithm is, optionally, applied to the intensities of the swipe contact prior to determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of: an unweighted sliding-average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and/or an exponential smoothing algorithm. In some circumstances, these smoothing algorithms eliminate narrow spikes or dips in the intensities of the swipe contact for purposes of determining a characteristic intensity.
The intensity of a contact on the touch-sensitive surface is, optionally, characterized relative to one or more intensity thresholds, such as a contact-detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and/or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform operations typically associated with clicking a button of a physical mouse or a trackpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform operations that are different from operations typically associated with clicking a button of a physical mouse or a trackpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact-detection intensity threshold below which the contact is no longer detected), the device will move a focus selector in accordance with movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise stated, these intensity thresholds are consistent between different sets of user interface figures.
An increase of characteristic intensity of the contact from an intensity below the light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes referred to as a “light press” input. An increase of characteristic intensity of the contact from an intensity below the deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes referred to as a “deep press” input. An increase of characteristic intensity of the contact from an intensity below the contact-detection intensity threshold to an intensity between the contact-detection intensity threshold and the light press intensity threshold is sometimes referred to as detecting the contact on the touch-surface. A decrease of characteristic intensity of the contact from an intensity above the contact-detection intensity threshold to an intensity below the contact-detection intensity threshold is sometimes referred to as detecting liftoff of the contact from the touch-surface. In some embodiments, the contact-detection intensity threshold is zero. In some embodiments, the contact-detection intensity threshold is greater than zero.
In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a respective press input or in response to detecting the respective press input performed with a respective contact (or a plurality of contacts), where the respective press input is detected based at least in part on detecting an increase in intensity of the contact (or plurality of contacts) above a press-input intensity threshold. In some embodiments, the respective operation is performed in response to detecting the increase in intensity of the respective contact above the press-input intensity threshold (e.g., a “down stroke” of the respective press input). In some embodiments, the press input includes an increase in intensity of the respective contact above the press-input intensity threshold and a subsequent decrease in intensity of the contact below the press-input intensity threshold, and the respective operation is performed in response to detecting the subsequent decrease in intensity of the respective contact below the press-input threshold (e.g., an “up stroke” of the respective press input).
In some embodiments, the display of representations 578A-578C includes an animation. For example, representation 578A is initially displayed in proximity of application icon 572B, as shown in
In some embodiments, display controller 588 causes the various user interfaces of the disclosure to be displayed on display 594. Further, input to device 580 is optionally provided by remote 590 via remote interface 592, which is optionally a wireless or a wired connection. In some embodiments, input to device 580 is provided by a multifunction device 591 (e.g., a smartphone) on which a remote control application is running that configures the multifunction device to simulate remote control functionality, as will be described in more detail below. In some embodiments, multifunction device 591 corresponds to one or more of device 100 in
In some embodiments, the device employs intensity hysteresis to avoid accidental inputs sometimes termed “jitter,” where the device defines or selects a hysteresis intensity threshold with a predefined relationship to the press-input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press-input intensity threshold or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press-input intensity threshold). Thus, in some embodiments, the press input includes an increase in intensity of the respective contact above the press-input intensity threshold and a subsequent decrease in intensity of the contact below the hysteresis intensity threshold that corresponds to the press-input intensity threshold, and the respective operation is performed in response to detecting the subsequent decrease in intensity of the respective contact below the hysteresis intensity threshold (e.g., an “up stroke” of the respective press input). Similarly, in some embodiments, the press input is detected only when the device detects an increase in intensity of the contact from an intensity at or below the hysteresis intensity threshold to an intensity at or above the press-input intensity threshold and, optionally, a subsequent decrease in intensity of the contact to an intensity at or below the hysteresis intensity, and the respective operation is performed in response to detecting the press input (e.g., the increase in intensity of the contact or the decrease in intensity of the contact, depending on the circumstances).
For ease of explanation, the descriptions of operations performed in response to a press input associated with a press-input intensity threshold or in response to a gesture including the press input are, optionally, triggered in response to detecting either: an increase in intensity of a contact above the press-input intensity threshold, an increase in intensity of a contact from an intensity below the hysteresis intensity threshold to an intensity above the press-input intensity threshold, a decrease in intensity of the contact below the press-input intensity threshold, and/or a decrease in intensity of the contact below the hysteresis intensity threshold corresponding to the press-input intensity threshold. Additionally, in examples where an operation is described as being performed in response to detecting a decrease in intensity of a contact below the press-input intensity threshold, the operation is, optionally, performed in response to detecting a decrease in intensity of the contact below a hysteresis intensity threshold corresponding to, and lower than, the press-input intensity threshold.
As used herein, an “installed application” refers to a software application that has been downloaded onto an electronic device (e.g., devices 100, 300, and/or 500) and is ready to be launched (e.g., become opened) on the device. In some embodiments, a downloaded application becomes an installed application by way of an installation program that extracts program portions from a downloaded package and integrates the extracted portions with the operating system of the computer system.
As used herein, the terms “open application” or “executing application” refer to a software application with retained state information (e.g., as part of device/global internal state 157 and/or application internal state 192). An open or executing application is, optionally, any one of the following types of applications:
As used herein, the term “closed application” refers to software applications without retained state information (e.g., state information for closed applications is not stored in a memory of the device). Accordingly, closing an application includes stopping and/or removing application processes for the application and removing state information for the application from the memory of the device. Generally, opening a second application while in a first application does not close the first application. When the second application is displayed and the first application ceases to be displayed, the first application becomes a background application.
Attention is now directed towards embodiments of user interfaces (“UI”) and associated processes that are implemented on an electronic device, such as portable multifunction device 100, device 300, or device 500.
Users interact with electronic devices in many different manners, including entering text into the electronic device. In some embodiments, an electronic device provides a virtual keyboard (e.g., soft keyboard) which mimics the layout of a physical keyboard and allows a user to select the letters to input. The embodiments described below provide ways in which an electronic device accepts handwritten inputs from a handwriting input device (e.g., a stylus) and converts the handwritten input into font-based text (e.g., computer text, digital text, etc.). Enhancing interactions with a device reduces the amount of time needed by a user to perform operations, and thus reduces the power usage of the device and increases battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally referred to as a user of the device.
In some embodiments, as shown in
In
As shown in
In
In
In
In some embodiments, handwritten input 604-1 is converted to font-based text. In some embodiments, font-based text is text that is entered when using a traditional text entry system such as a physical keyboard or soft keyboard. In some embodiments, the text is formatted using a particular font style. For example, the font-based text is Times New Roman with 12 point size or Arial with 10 point size, etc. In some embodiments, handwritten input 604-3 is converted after a threshold amount of delay (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds). In some embodiments, handwritten input 604-3 is converted after the visual characteristics of handwritten input 604-3 is modified to indicate that handwritten input 604-3 will be converted (e.g., as described in
In some embodiments, the size of the handwritten input after it has been converted is the default font size for the text entry field. In some embodiments, the size of the handwritten input changes before handwritten input is converted into font-based text. In some embodiments, the size of the font-based text matches the size of the handwritten input and then the size of the font-based text is changed to match the default size for the text entry field (e.g., the size is changed after an animation changing the handwriting input to the font-based text). In some embodiments, the size changes during the animation from handwriting input to font-based text. In some embodiments, the animation of converting handwriting input to font-based text comprises morphing the handwriting input to font-based text. In some embodiments, the handwriting input is disassembled (e.g., into pieces or particles) and re-assembled as the font-based text (e.g., such as described below with respect to method 2000). In some embodiments, the handwriting input dissolves or fades out and the font-based text dissolves-in or fades in. In some embodiments, the handwriting input moves toward the final location of the font-based text (e.g., aligns itself with the text entry region or any pre-existing text) while dissolving and the font-based text concurrently appears while moving toward the final location. Thus, in some embodiments, the handwriting input and the font-based text can be simultaneously displayed on the display during at least part of the animation (e.g., to reduce the animation time).
In
In
In
In
In some embodiments, after the user lifts off stylus 203 from touch screen 504 for a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds), then device 500 analyzes, interprets, and converts the handwritten inputs into font-based text (e.g., handwritten input 604-5). In some embodiments, as described above, handwritten input 604-5 is entered into text entry field 602-4 instead of text entry field 602-3 because the user paused handwritten input for a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds) such that handwritten input 604-5 is not considered a continuation of handwritten input 604-3 or handwritten input 604-5 (e.g., which would optionally merit the handwritten input to be entered into text entry field 602-3). In some embodiments, concurrently with or after handwritten input 604-5 is converted into font-based text, text entry field 602-4 returns to its original size.
In
In some embodiments, pop-up 606 is displayed above handwritten input 604-6 or otherwise within the vicinity of handwritten input 604-6 (e.g., within 5 mm, 1 cm, 1.5 cm, 3 cm, etc.). In some embodiments, the word or letters associated with pop-up 606 are highlighted. In some embodiments, pop-up 606 includes the highest confidence interpretation of handwritten input 604-6 (e.g., “Salem”). In some embodiments, pop-up 606 includes more than one potential interpretation of handwritten input 604-6 (e.g., corresponding to one or more selectable options). In some embodiment, pop-up 606 is selectable to cause the conversion of handwritten input 604-6 into the selected interpretation (e.g., as opposed to converting after a threshold time delay or other time-based heuristic). In some embodiments, pop-up 606 is displayed after the user has lifted off stylus 203 from touch screen 504 and device 600 has had a chance to analyze and interpret the entire handwritten sequence (e.g., the entire word, the entire sentence, the sequence of letters, etc.). In some embodiments, pop-up 606 is displayed at any time while the user is performing handwritten input and is updated as the user writes additional letters that is recognized by device 500. For example, pop-up 606 optionally initially appears after the user has written “Sa” and displays “Sa”. In such examples, after the user writes “1”, then pop-up 606 is updated to display “Sal”. In some embodiments, after the user writes “em”, then pop-up 606 is updated to display “Salem” (e.g., in some embodiments, the pop-up is updated with new letters after each letter or after several letters). In some embodiments, pop-up 606 is displayed regardless of the confidence level of the interpretation of the handwritten input (e.g., pop-up 606 is optionally always displayed and provides the user a method in which to “accept” the suggested font-based text and cause conversion of handwritten input into the suggested font-based text without regard to timers that are being used to determine when to convert handwritten text into font-based text). In some embodiments, pop-up 606 includes a selectable option to reject the suggestion or otherwise dismiss pop-up 606. In some embodiments, dismissing the pop-up or rejecting the suggestion does not cause handwritten input 604-6 to never be converted. In some embodiments, dismissing the pop-up or rejecting the suggestion causes handwritten input 604-6 to not be converted at that point in time, but handwritten input 604-6 is still optionally converted at a later point in time based on other heuristics, such as the timer-based conversion heuristics.
As shown in
In some embodiments, based on the confidence level of device 500 in the written letters in handwritten input 604-6, the converted font-based text is placed in displayed in different locations in the text entry field. For example, if the confidence level of device 500 is below a threshold level (e.g., 25% confidence, 50% confidence, 75% confidence, etc.), then the converted font-based text is not aligned with any pre-existing text or the text entry field. Instead, in some embodiments, the converted font-based text is left in the same position as the original handwritten input indicating to the user that device 500 is not confident in the conversion. In some embodiments, if the confidence level is above the threshold level, then the converted font-based text is aligned with any pre-existing text in the text entry field or left-aligned with the text entry field (e.g., if there is no pre-existing text).
In
In
In
In
In
In
In
As described below, the method 700 provides ways to convert handwritten inputs into font-based text. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, an electronic device (e.g., an electronic device, a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device) including a touch screen, or a computer including a touch screen, such as device 100, device 300, device 500, device 501, or device 591) in communication with a touch-sensitive displays (702), on the touch-sensitive display, a user interface including a first text entry region, such as in
In some embodiments, while displaying the user interface, the electronic device receives (704), via the touch-sensitive display, a user input comprising a handwritten input directed to the first text entry region, such as in
In some embodiments, while receiving the user input, the electronic device displays (706) a representation of the handwritten input in the user interface at a location corresponding to the text entry region, such as in
In some embodiments, after displaying the representation of the handwritten input in the user interface (708), such as in
In some embodiments, the replacement occurs while the input is received (e.g., the first part of the handwritten input is replaced while the user is still inputting the second part of the handwritten input). In some embodiments, the replacement occurs after the input ends (e.g., after a threshold amount of time without receiving handwritten input, after the user completes writing a word or sentence, or after satisfaction of some other input termination criteria). In some embodiments, the replacement occurs after displaying proposed text to the user and receiving an input selecting or confirming proposed text.
In some embodiments, the system determines the letters and/or words that the user wrote in the handwritten input and converts them into computerized text. For example, the handwritten input is optionally replaced with text with 12-point Times New Roman font (e.g., or other suitable font). In some embodiments, font-based text is 10-point sized, 12-point sized, etc. and optionally is Arial, Calibri, Times New Roman, etc. In some embodiments, the computerized text (e.g., font-based text) replaces the handwritten input. In some embodiments, the font-based text is displayed before or after the portion of the handwritten input is removed from display (e.g., 0.5 seconds before or after, 1 second before or after, 3 seconds before or after, etc.). In some embodiments, an animation is shown converting the handwritten input into the computerized text or otherwise removing the handwritten input and displaying the computerized text. In some embodiments, the location of the computerized text overlaps with the location where the handwritten input existed before the conversion. In some embodiments, the computerized text is a smaller size than the handwritten input (e.g., the font size is smaller than the handwritten input). In some embodiments, the handwritten input is converted into font-based text that has the same size as the handwritten input (e.g., the size of the font-based text is matched to the handwritten input) before the font-based text is then updated to its final size (e.g., the default size of the font-based text or the default size of the text entry region). In some embodiments, the size of the handwritten input is modified to the final size of the font-based text (e.g., the default size of the font-based text or the default size of the text entry region) before the handwritten input is converted to font-based text (e.g., in its final size—which matches the final size of the handwritten input). In some embodiments, the size of the handwritten input is not changed and the font-based text appears already in its final size without matching the size of the handwritten input and without changing from an initial size to the final size. Similarly, in some embodiments, the location of the text is optionally updated before or after the conversion. In some embodiments, the handwritten input is moved to the final location before conversion, the font-based text appears (e.g., when it is converted) at the location of the handwritten input before moving to its final location, or the font-based text appears (e.g., when it is converted) at the final location without an animation moving the font-based text from an initial position to the final position. In some embodiments, the animation includes any combination of (e.g., and in any order) changing size and/or location of the handwritten input or font-based text to result in the final location and size from the initial location and size of the handwritten input. In some embodiments, regardless of the size of the user's writing, the representation of the handwritten text is displayed at the final size of the font-based text (e.g., the default size of the font-based text or the default size of the text entry region). In some embodiments, as a result of the conversion operation, the font-based text is provided to the text entry or text entry region as a text input. In some embodiments, the animation of the handwritten text converting into font-based text is similar to or shares similar features as the conversion of handwritten input into font-based text described below with respect to method 2000. In some embodiments, when the handwritten input is converted into font-based text, an animation is displayed of the handwritten input dissolving into particles and moving to the location where the font-based location appears similar to the animation described below with respect to method 2000 (e.g., and/or described below with respect to
In some embodiments, after displaying the representation of the handwritten input in the user interface (708), such as in
The above-described manner of converting handwritten inputs to text (e.g., by receiving the input at or near a text entry field and replacing the handwritten input with text if certain criteria are satisfied) allows the electronic device to provide the user with the ability to write directly onto a user interface to enter text (e.g., by accepting handwritten inputs and automatically determining the text that corresponds to the handwritten input and entering the text into the respective text entry field), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to handwrite text directly onto a touch screen display without requiring the user to select a respective text field and then use a keyboard (e.g., physical or virtual keyboard) to enter text into the text field), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the text entry region occurs while continuing to receive the handwritten input (714), such as in
The above-described manner of converting handwritten inputs to text (e.g., by displaying the font-based text while continuing to receive handwritten input) allows the electronic device to provide the user with the ability to receive instant feedback of the text that the user is writing (e.g., by accepting handwritten inputs and converting the handwritten inputs into text while the user is still continuing to provide handwritten inputs), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to verify that the conversion is correct without needing to wait until all of the input is converted at once or perform a separate input to trigger conversion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the text entry region occurs in response to detecting a pause for longer than a time threshold (e.g., 0.5, 1, 2, 3, 5 seconds) in the handwritten input (716), such as in
The above-described manner of converting handwritten inputs to text (e.g., by displaying the font-based text after a pause in the handwritten input) allows the electronic device to convert handwritten text without unnecessarily distracting the user (e.g., by converting the handwritten text after the user has paused the handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to complete his or her current input before performing the conversion, which reduces the chances of distracting the user, while improving the accuracy of the conversion and balances providing the user with feedback on the user's handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, after displaying the representation of the handwritten input in the user interface, the electronic device concurrently displays (718), on the touch-sensitive display, such as in
In some embodiments, ceasing to display the at least the portion of the representation of the handwritten input and displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the text entry region occurs in response to detecting selection of the selectable option (726), such as in
The above-described manner of presenting a handwriting conversion option to the user (e.g., by displaying a selectable option to convert the handwritten text) allows the electronic device to present the user with the option of whether to convert the handwritten text and what to convert the handwritten text to (e.g., by converting the handwritten text when the user selects the selectable option to acknowledge the conversion), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to visually verify the conversion and acknowledge and/or confirm the conversion without requiring the user to verify the conversion after the conversion and then making any required edits if the conversion is incorrect), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the text entry region comprises a text entry field (728), such as in
The above-described manner of entering the font-based text (e.g., by converting and entering the font-based text into a text entry field) allows the electronic device to enter the user's handwritten input into an appropriate text field (e.g., by converting the handwritten text and displaying the font-based text into a text entry field that accepts font-based text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by entering the converted text into the appropriate text field without requiring the user to precisely provide handwriting input in the desired text entry field and without requiring the user to separately move the converted text into a text entry field after conversion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the at least the portion of the handwritten input includes handwritten input detected inside a boundary of the text entry region and handwritten input detected outside of the boundary of the text entry region (730), such as in
The above-described manner of accepting handwritten input (e.g., by recognizing handwritten input that is both inside a text entry region and outside a text entry region as directed to the text entry region) allows the electronic device to provide the user with compatibility with natural handwriting characteristics (e.g., by accepting handwritten text that potentially extends outside of a text entry region and is not fully within a text entry region), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by accepting natural handwriting inputs that may be large and extend outside of a given text entry region without requiring the user to perfectly write within a given text entry region for the handwritten input to be accepted), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, handwritten input detected within a margin of error region, larger than the text entry region and surrounding the text entry region, is eligible to be converted to font-based text in the text entry region, and handwritten input detected outside of the margin of error region is not eligible to be converted to font-based text in the text entry region (732), such as in
The above-described manner of accepting handwritten input (e.g., by providing a margin of error area around a text entry region in which handwritten input is eligible to be converted to font-based text) allows the electronic device to provide the user with compatibility with natural handwriting characteristics (e.g., by accepting handwritten text that potentially extends outside of a text entry region and is not fully within a text entry region), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by accepting natural handwriting inputs that may be large and extend outside of a given text entry region without requiring the user to perfectly write within a given text entry region for the handwritten input to be accepted), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the electronic device receives (734), via the touch-sensitive display, a second user input comprising a handwritten input directed to a second text entry region in the user interface, such as in
In some embodiments, after receiving the second user input (736), in accordance with a determination that the second user input satisfies one or more second criteria, including a criterion that is satisfied when the second user input is detected within a time threshold of the user input, the electronic device displays (738) font-based text corresponding to the second user input in the text entry region, such as in
In some embodiments, after receiving the second user input (736), in accordance with a determination that the second user input does not satisfy the one or more second criteria, the electronic device displays (740) font-based text corresponding to the second user input in the second text entry region, such as in
The above-described manner of converting handwritten input (e.g., by entering subsequent handwritten inputs into a given text entry region even if the subsequent handwritten input is directed to another text entry region) allows the electronic device to provide the user with compatibility with natural handwriting characteristics (e.g., by accepting continued handwritten text that is fully outside of a given text entry region and potentially directed to another text entry region as long as the continued handwritten text is within a certain time threshold from the previous handwritten text that is directed to the given text entry region), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by accepting natural handwriting inputs without requiring the user to pause his or her handwritten input and reposition the handwritten input to the desired text entry region or separately moving converted text from the second text entry region to the text entry region after conversion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more second criteria include a criterion that is satisfied when a majority of the second user input is directed to the text entry region rather than the second text entry region, such as in
The above-described manner of converting handwritten input (e.g., by entering subsequent handwritten inputs into a given text entry region if a majority of the subsequent handwritten input is directed to the given text entry region rather than another text entry region) allows the electronic device to provide the user with compatibility with natural handwriting characteristics (e.g., by accepting continued handwritten text that extends outside of a given text entry region if a majority of the continued handwritten text is within the given text entry region), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by continued natural handwriting inputs without requiring the user to pause his or her handwritten input and reposition the handwritten input to the desired text entry region or separately moving converted text from the second text entry region to the text entry region after conversion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the text entry region includes (744), such as in
The above-described manner of displaying font-based text (e.g., by displaying the font-based text with a first visual characteristic before committing the text to the text entry field and by displaying the font-based text with a second visual characteristic after committing the text to the text entry field) allows the electronic device to provide the user with feedback on the progress of converting the user's handwritten text (e.g., by displaying the font-based text with a first visual characteristic before committing and a second visual characteristic after committing the font-based text to the text entry region), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with visual feedback on the progress of converting handwritten input to font-based text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the text entry region includes (750), such as in
The above-described manner of providing visual feedback (e.g., by displaying the font-based text with a first visual characteristic if the confidence in the interpretation and conversion is at a first level and by displaying the font-based text with a second visual characteristic if the confidence in the interpretation and conversion is at a second level) allows the electronic device to provide the user with visual feedback of the confidence and/or accuracy of the conversion, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with a visual cue of the confidence level of the conversion of the user's handwritten user input, thus providing the user with an indication of whether to confirm that the conversion is accurate), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the text entry region includes (756), such as in
The above-described manner of displaying font-based text (e.g., by displaying the font-based text at a location based on the confidence level of the conversion of the text from handwritten input) allows the electronic device to provide the user with visual feedback of the confidence and/or accuracy of the conversion, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with a visual cue of the confidence level of the conversion of the user's handwritten user input by not moving the font-based text into its final location, thus providing the user with an indication of whether to confirm that the conversion is accurate), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, such as in
The above-described manner of converting handwritten input (e.g., by converting the handwritten text based on a number of different factors) allows the electronic device to select the most appropriate time to convert handwritten text based on the situation (e.g., by converting text based on timing of the input, context, punctuation, distance and angle of the stylus, inputs interacting with other elements, etc.), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by converting text at a time that is least intrusive to the user while balancing the speed to convert the text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while receiving the user input, in accordance with a determination that one or more second criteria are satisfied, the electronic device moves (764) at least a portion of the representation of the handwritten input in the user interface to reveal space in the user interface for receiving additional handwritten input, such as in
The above-described manner of receiving handwritten input (e.g., by moving previous handwritten input as handwritten input is received to provide room for more handwritten input) allows the electronic device to provide the user with space to provide handwritten input (e.g., by spatially moving previously inputted handwritten input to provide room for receiving further handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to continue providing handwritten input without having to reset the location of the user's input to ensure that it stays within the text entry region), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while receiving the user input, in accordance with a determination that one or more third criteria are satisfied, the electronic device expands (766) a boundary of the text entry region to create space in the text entry region for receiving additional handwritten input, such as in
The above-described manner of receiving handwritten input (e.g., by expanding the size of the text entry region) allows the electronic device to provide the user with space to provide handwritten input (e.g., by expanding the text entry region horizontally and/or vertical when the user begins to reach the boundary of the text entry region to provide room for receiving further handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to continue providing handwritten input into the text entry region), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, expanding the boundary of the text entry region, such as in
In some embodiments, expanding the boundary of the text entry region, such as in
The above-described manner of receiving handwritten input (e.g., by expanding the boundaries of the text entry region based on the location of the text entry region on the screen) allows the electronic device to provide the user with space to provide handwritten input (e.g., by moving a respective boundary of the text entry region based on the location of the text entry region to provide the most natural location to perform handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with space in which to comfortably and naturally perform handwritten input without requiring the user to write in an awkward location), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, displaying the representation of the handwritten input in the user interface while receiving the user input includes displaying an animation of one or more visual characteristics of the representation of the handwritten input changing as a function of elapsed time since the corresponding handwritten input was received (774), such as in
The above-described manner of displaying handwritten input (e.g., by changing the visual characteristics of the handwritten input over time) allows the electronic device to provide the user with a visual cue of how long since the handwritten input has been received and how long the handwritten input has been processed (e.g., by displaying an animation of the handwritten input changing visual characteristics based on how the time since receiving the handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with a visual indication of the elapsed time since the handwritten input was received), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, ceasing to display the at least the portion of the representation of the handwritten input and displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the text entry region includes displaying an animation of the representation of the handwritten input morphing into the font-based text (776), such as in
The above-described manner of displaying handwritten input (e.g., by displaying an animation of the handwritten input morphing into the font-based text) allows the electronic device to provide the user with a visual cue that the handwritten input is converted into the font-based text (e.g., by displaying an animation of the handwritten input morphing into the font-based text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with a visual indication that it is the user's handwritten input that is being processed, interpreted, and converted into the font-based text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the at least the portion of the handwritten input corresponds to font-based text that includes a typographical error, and displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the text entry region includes displaying the font-based text with the typographical error having been corrected (778), such as in
The above-described manner of converting handwritten input (e.g., by removing typographical errors when converting handwritten input to font-based text) allows the electronic device to automatically provide the user with an error-free font-based text (e.g., by automatically removing typographical errors when converting handwritten input to font-based text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically removing typographical errors for the user without requiring the user to separately determine whether a typographical error exists and to perform additional inputs to edit the font-based text and remove the typographical error), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, after displaying the representation of the handwritten input in the user interface (780), in accordance with the determination that the user input satisfies one or more first criteria (782), the electronic device transmits (784) the font-based text corresponding to the at least the portion of the representation of the handwritten input to a second electronic device, separate from the electronic device, such as in
The above-described manner of transmitting text to a second electronic device (e.g., by receiving handwritten input on the electronic device, converting it into font-based text, and transmitting the font-based text to the second electronic device) allows the electronic device to provide the user with a handwritten entry method of entering text on a second electronic device (e.g., by receiving handwritten input from the user, converting the handwritten input to font-based text and transmitting text to the second electronic device), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by accepting the user's handwritten input and transmitting the font-based text to the second electronic device without requiring the user to use a virtual keyboard or use a traditional remote control to enter text on the second electronic device), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the second electronic device is displaying a user interface that includes one or more respective text entry regions, including a respective text entry region that corresponds to the text entry region displayed by the electronic device (786), such as in
In some embodiments, the electronic device detects, at the electronic device, the one or more respective text entry regions displayed by the second electronic device (788), such as in
In some embodiments, transmitting the font-based text corresponding to the at least the portion of the representation of the handwritten input to the second electronic device includes transmitting the font-based text to the respective text entry region on the second electronic device (792), such as in
The above-described manner of transmitting text to a second electronic device (e.g., by displaying the same text entry regions on the electronic device as is being displayed on the second electronic device) allows the electronic device to provide the user with an intuitive interface by which to transmit text to the second electronic device (e.g., by mirroring the user interface of the second electronic device to the electronic device and transmitting text from the electronic device to the appropriate text entry region on the second electronic device), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the same user interface on the electronic device as is shown on the first electronic device so that the user can easily and intuitively select which text entry region to enter text into, without requiring the user to perform additional inputs or use a traditional remote control to select which text entry region to enter text into), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the text entry region is a multi-line text entry region, and the font-based text corresponding to the at least the portion of the representation of the handwritten input is displayed in a first line of the multi-line text entry region (794), such as in FIG. AA (e.g., the text entry region supports multiple lines of text).
In some embodiments, while displaying the font-based text corresponding to the at least the portion of the representation of the handwritten input in the first line of the multi-line text entry region, the electronic device receives (796), via the touch-sensitive display, a second user input comprising a handwritten input directed to the first text entry region, such as in
In some embodiments, after receiving the second user input (798), in accordance with a determination that one or more second criteria are satisfied, the electronic device displays (798-2) font-based text corresponding to the second user input in a second line, different than the first line, of the multi-line text entry region, such as in
In some embodiments, after receiving the second user input (798), in accordance with a determination that one or more second criteria are satisfied, in accordance with a determination that the one or more second criteria are not satisfied, the electronic device displays (798-4) the font-based text corresponding to the second user input in the first line of the multi-line text entry region, such as in
The above-described manner of entering handwritten text (e.g., by entering the text into a second line of a text entry region that supports multiple lines of text when the user input indicates a request to enter text in a second line) allows the electronic device to provide the user with an intuitive method of entering multi-line text (e.g., by entering text in a second line of the text entry region if certain criteria for the handwritten input are met), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by determining whether a new line should be created and entering text into the new line, without requiring the user to perform additional user inputs or wait until after the handwritten text is converted to manually edit the font-based text to insert line breaks at the desired locations), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more second criteria are satisfied when the second user input is detected more than a threshold distance below the user input (e.g., 6 points, 12 points, 18 points, 20 points, 24 points, etc.), and the one or more second criteria are not satisfied when the second user input is detected less than the threshold distance below the user input (798-6), such as in
The above-described manner of entering multi-lined handwritten text (e.g., by entering the text into a second line of a text entry region when a user input is received that is more than a threshold distance below the previous line of text indicating a request to enter text in a second line) allows the electronic device to provide the user with an intuitive method of entering multi-line text (e.g., by accepting handwritten text below the previous line of text and interpreting the input as a request to enter the handwritten text into a line below the previous line of text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by entering text into a new line when handwritten text is received a threshold distance below the previous line of text, without requiring the user to perform additional user inputs or wait until after the handwritten text is converted to manually edit the font-based text to insert line breaks at the desired locations), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more second criteria are satisfied when the second user input includes a stylus input detected at the second line in the multi-line text entry region, and the one or more second criteria are not satisfied when the second user input does not include a stylus input detected at the second line in the multi-line text entry region (798-8), such as in
The above-described manner of entering multi-lined handwritten text (e.g., by receiving a tap at a second line indicating a request to enter text in a second line and inserting the text into a second line of a text entry region) allows the electronic device to provide the user with an intuitive method of entering multi-line text (e.g., by accepting a gestural input below the previous line of text and interpreting the input as a request to enter the handwritten text into a line below the previous line of text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by entering text into a new line when receiving a tap below the previous line of text, without requiring the user to perform additional user inputs or wait until after the handwritten text is converted to manually edit the font-based text to insert line breaks at the desired locations), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, a selectable option for moving to the second line is displayed concurrently with the font-based text corresponding to the at least the portion of the representation of the handwritten input, the one or more second criteria are satisfied when the selectable option has been selected, and the one or more second criteria are not satisfied when the selectable option has not been selected (798-10), such as in
The above-described manner of entering multi-lined handwritten text (e.g., by receiving a selection on a selectable option for inserting a new line of text below the previous line of text) allows the electronic device to provide the user with an easy method of entering multi-line text (e.g., by providing a selectable option that is selectable to insert handwritten text into a line below the previous line of text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing a selectable option to enter a new line of text and entering text into a new line in response to receiving a selection of the selectable option, without requiring the user to manually edit the font-based text to insert line breaks at the desired locations after the handwritten text has been converted into font-based text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the electronic device receives (798-12), via the touch-sensitive display, a second user input, such as in
In some embodiments, in response to receiving the second user input (798-14), in accordance with a determination that the second user input is detected in a region of the user interface not corresponding to a text entry region, the electronic device performs (798-18) a scrolling operation in the user interface based on the second user input, such as in FIG. 6Y (e.g., if the user input is not directed to a text entry region, then do not interpret the user input as a request to insert text). For example, if the user interacts with another user element that is not a text entry region, then do not perform handwritten conversion processes. In some embodiments, for example, if the user performs a scrolling or other type of navigation gesture, then perform the navigation according to the user input instead of inserting font-based text based on handwritten input.
The above-described manner of interpreting user input (e.g., by interpreting input as handwritten text when it is received in a text entry region, but not interpreting the input as handwritten text if it is not received in a text entry region) allows the electronic device to provide the user with an easy method of entering text (e.g., by allowing the user to interact with the device in a non-text-method if the input does not indicate a request to enter text but also accepting handwritten input if the input indicates a request to enter text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically determining whether the user is request to enter text or to otherwise interact with the user interface without requiring the user to perform additional inputs to switch to text-entry mode or to interact with a separate user interface or use a separate device to enter text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the animation of the representation of the handwritten input morphing into the font-based text includes (798-20): in accordance with a determination that the text entry region does not yet include font-based text, animating the representation of the handwritten input morphing (e.g., directly) into font-based text at a final location in the text entry region and at a final size at which the font-based text is going to be displayed (798-22), such as in
The above-described manner of converting handwritten inputs to text (e.g., by displaying an animation of the handwritten input concurrently changing to the final size of the font-based text and moving to the final location) allows the electronic device to provide the user with a visual cue that the handwritten input is converted into the font-based text (e.g., by displaying an animation of the handwritten input morphing into the font-based text in one step), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with a visual indication that it is the user's handwritten input that is being processed, interpreted, and converted into the font-based text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the animation of the representation of the handwritten input morphing into the font-based text includes (798-24): in accordance with a determination that the text entry region does not yet include font-based text, animating the representation of the handwritten input morphing into font-based text at an intermediate size based on a size of the representation of the handwritten input, and subsequently animating the font-based text at the intermediate size morphing into font-based text at a final location in the text entry region and at a final size, different than the intermediate size, at which the font-based text is going to be displayed (798-26), such as in
The above-described manner of converting handwritten inputs to text (e.g., by displaying an animation of the handwritten input first converting into a font-based text with an intermediate size (between the final size and the size of the handwritten input) and then converting from the intermediate size into the final size while moving to the final location) allows the electronic device to provide the user with a visual cue that the handwritten input is converted into the font-based text (e.g., by displaying an animation of the handwritten input morphing into the font-based text in two steps to emphasize that the process is both converting the handwritten input into font-based text and resizing and moving the font-based text into the proper size and position), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient, which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the animation of the representation of the handwritten input morphing into the font-based text includes (798-28): in accordance with a determination that the text entry region does include previously-entered font-based text (e.g., font-based text that is displayed in the text entry region before the handwritten input is converted to font-based text (e.g., the font-based text corresponding to the handwritten input will be added to the pre-existing font-based text in the text entry region)), animating the representation of the handwritten input morphing into font-based text at an intermediate size based on a size of the representation of the handwritten input, and subsequently animating the font-based text at the intermediate size morphing into font-based text at a final location in the text entry region and at a final size, different than the intermediate size, at which the font-based text is going to be displayed, wherein the final size of the font-based text corresponding to the handwritten input is the same as a size of the previously-entered font-based text (798-30), such as in
The above-described manner of converting handwritten inputs to text (e.g., by displaying an animation of the handwritten input first converting into a font-based text with an intermediate size (between the final size and the size of the handwritten input) and then converting from the intermediate size into the same size as any pre-existing text while moving to the final location (e.g. aligned with the pre-existing text)) allows the electronic device to provide the user with a visual cue that the handwritten input is converted into the font-based text (e.g., by displaying an animation of the handwritten input morphing into the font-based text in two steps to emphasize that the process is both converting the handwritten input into font-based text and resizing and moving the font-based text into the proper size and position), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient, which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
Users interact with electronic devices in many different manners, including entering text into the electronic device. In some embodiments, an electronic device displays text in a text field or a text region. The embodiments described below provide ways in which an electronic device selects and/or deletes text using a handwriting input device (e.g., a stylus). Enhancing interactions with a device reduces the amount of time needed by a user to perform operations, and thus reduces the power usage of the device and increases battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally referred to as a user of the device.
In
In
In
In
In some embodiments, as shown in
In
In
In
In
In
In
In
In
In
In
It is understood that the above-described deletion and selection gestures can be applied on a per-letter basis or a per-word basis. In other words, if a gesture is received on one or more letters of a word, then in some embodiments, only those one or more letters are subject to the respective selection or deletion command. In some embodiments, if a gesture is received on one or more letters of a word, then the entire word associated with the one or more letters is subject to the respective selection or deletion command.
In
In some embodiments, in response to receiving the handwritten input (optionally in response to a lift-off corresponding to the handwritten input (e.g., lift-off of stylus 203) and optionally after a threshold amount of time, such as 0.5 seconds, 1 second, 3 seconds, 5 seconds, etc.), the selected word “woke” is replaced with the characters corresponding to the handwritten input, as shown in
As described below, the method 900 provides ways to interpret handwritten inputs to select or delete text. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, an electronic device (e.g., an electronic device, a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device) including a touch screen, or a computer including a touch screen, such as device 100, device 300, device 500, device 501, or device 591) in communication with a touch-sensitive display displays (902), on the touch-sensitive display, a user interface including a first editable text string that includes one or more text characters, such as in
In some embodiments, while displaying the user interface, the electronic device receives (904), via the touch-sensitive display, a user input comprising a handwritten input corresponding to a line drawn through multiple text characters in the first editable text string, such as in
In some embodiments, in response to receiving the user input (906), in accordance with a determination that the handwritten input satisfies one or more first criteria, the electronic device initiates (908) a process to select the multiple text characters of the first editable text string, such as in
In some embodiments, in response to receiving the user input (906), in accordance with a determination that the handwritten input satisfies one or more second criteria, different than the first criteria, the electronic device initiates (910) a process to delete the multiple text characters of the first editable text string, such as in
The above-described manner of selecting or deleting text (e.g., by receiving a handwritten user input on editable text and interpreting the handwritten user input as a selection or deletion based on the characteristics of the input) allows the electronic device to provide the user with the ability to edit text (e.g., by accepting handwritten inputs and automatically determining whether the uses intends to select text or delete text based on the input gestures), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to use a handwritten input to either select and delete text without requiring the user to navigate to a separate user interface or menu to activate the selection function or the deletion function), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, initiating the process to select the multiple text characters of the first editable text string includes displaying a representation of the line corresponding to the handwritten input with the multiple text characters in the first editable text string (912), such as in
The above-described manner of selecting (e.g., by displaying the user's input as the user is inputting it) allows the electronic device to provide the user with feedback on what characters the user is requesting to be selected (e.g., by providing a visual indication of where and what the user is interacting with), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by giving the user feedback on what characters are being identified for selection or deletion without requiring the user to guess or perform additional inputs to correct any errors in selection or deletion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, while displaying the representation of the line corresponding to the handwritten input with the multiple text characters in the first editable text string, the electronic device receives (914), via the touch-sensitive display, an input corresponding to selection of the line, such as in
In some embodiments, in response to receiving the input corresponding to the selection of the line, the electronic device causes (916) the multiple text characters in the first editable text string to be selected for further action, such as in
The above-described manner of selecting text (e.g., by displaying the user's input underlining the multiple characters that were selected to be highlighted and highlighting the words after receiving the user's selection of the line) allows the electronic device to provide the user with feedback on what characters the user is requesting to be selected (e.g., by providing a visual indication of what characters would be selected and giving the user the opportunity to confirm the selection), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with the opportunity to confirm what characters would be selected or providing the user an opportunity to exit from selection mode without requiring the user to perform additional inputs to correct errors in selection or exit selection mode), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, initiating the process to select the multiple text characters of the first editable text string includes selecting the multiple text characters in the first editable text string without displaying a representation of the line corresponding to the handwritten input with the multiple text characters (918), such as in
The above-described manner of selecting text (e.g., by selecting the multiple characters as the user is performing the selection input gesture) allows the electronic device to provide the user with feedback on what characters the user is requesting to be selected (e.g., by providing a visual indication of what characters would be selected), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with the opportunity to see the selection occurring as the user is performing the input to confirm that the intended characters are being selected without requiring the user to perform additional inputs to correct errors in selection), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, initiating the process to delete the multiple text characters of the first editable text string includes displaying the multiple text characters with a first value for a visual characteristic, and displaying a remainder of the first editable text string with a second value, different than the first value, for the visual characteristic while the user input is being received (920), such as in
The above-described manner of deleting text (e.g., by changing the visual characteristics of the characters that have been selected by the user for deletion so far) allows the electronic device to provide the user with feedback on what characters the user is requesting to be deleted (e.g., by providing a visual indication of what characters would be deleted), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with the opportunity to see what characters would be deleted as the user is performing the input to confirm that the intended characters will be deleted without requiring the user to perform additional inputs to correct errors in deletion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, while displaying the multiple text characters with the first value for the visual characteristic, and displaying the remainder of the first editable text string with the second value for the visual characteristic, the electronic device detects (922) liftoff of the user input, such as in
The above-described manner of deleting text (e.g., by performing the deletion after the user has lifted off from interacting with the touch screen) allows the electronic device to provide the user with the ability to confirm the text to be deleted before performing the deletion (e.g., by not deleting the text when the user performs the deletion gesture, but allowing the user to verify the text to be deleted and deleting the text after the user has lifted off, indicating confirmation of the deletion), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with the opportunity to see what characters would be deleted to confirm that the intended characters will be deleted before lifting off to perform the deletion without requiring the user to perform additional inputs to correct errors in deletion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, before detecting the liftoff of the user input, the electronic device displays (926), with the first editable text string, a representation of the line corresponding to the handwritten input, such as in
The above-described manner of deleting text (e.g., by removing the display of the handwritten input at the time that the deletion is performed) allows the electronic device to clear the display of executed gestures (e.g., by removing the representation of the deletion gesture at the time that the deletion is executed or after the deletion is executed), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with multiple visual indications that the deletion has been performed including removing the residual handwritten gesture), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, after initiating the process to delete the multiple text characters of the first editable text string (930), in accordance with a determination that the handwritten input extends more than a threshold distance (e.g., 0.5 cm, 1 cm, 2 cm, 5 cm) away from the multiple text characters of the first editable text string, the electronic device cancels (932) the process to delete the multiple text characters of the first editable text string, such as in
The above-described manner of canceling deletion of text (e.g., by interpreting the user's gesture extending the input away from the text characters by a certain threshold distance as a request to cancel the deletion function) allows the electronic device to provide the user with the opportunity to cancel deleting text (e.g., by accepting input that extends away from the characters that have been marked for deletion as a request to cancel the deletion process), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with an opportunity to cancel the deletion function without requiring the user to re-enter all of the text that the user was not intending to delete), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while receiving the user input, the electronic device displays (934), with the first editable text string, a representation of the line corresponding to the handwritten input with a first value for a visual characteristic, such as in
The above-described manner of deleting text (e.g., by changing the visual characteristics of the representation of the user's handwriting input) allows the electronic device to provide the user with feedback that the user's input has been properly interpreted as a request to delete text (e.g., by providing a visual indication that the user's input gesture has been processed and interpreted as a deletion request), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with feedback at the time at which the user's input is recognized and interpreted as a deletion request and providing the user with the visual feedback that the characters over which the gesture is overlapping would be deleted), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, initiating the process to delete the multiple text characters of the first editable text string includes deleting the multiple text characters of the first editable text string (940), such as in
The above-described manner of providing a deletion undo function (e.g., by displaying a selectable option for undoing the deletion) allows the electronic device to provide the user with the option to undo the deletion (e.g., by providing a selectable option that is selectable to undo the deletion), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with the option to undo the deletion without requiring the user to manually re-enter all of the text that was deleted), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, initiating the process to select the multiple text characters of the first editable text string includes selecting the multiple text characters of the first editable text string (944), such as in
The above-described manner of providing function related to the selected text (e.g., by displaying a user interface with selectable options to perform certain functions to or with the selected text) allows the electronic device to provide the user with options for interacting with the selected text (e.g., by, after selecting the selected text, displaying one or more selectable options for performing one or more functions, respectively, on the selected text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically providing the user with functions to perform on the selected text without requiring the user to perform additional inputs or navigate to a separate user interface to perform the same functions), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the process to select the multiple text characters of the first editable text string includes selecting the multiple text characters of the first editable text string before detecting liftoff of the user input (948), such as in
The above-described manner of selecting and deleting text (e.g., by performing the selection functions before detecting a liftoff, but performing the deletion function after detecting liftoff) allows the electronic device to perform the selection or deletion at the appropriate time (e.g., by performing selection while receiving the selection gesture but performing the deletion after the user has had a chance to confirm the text that the user wants to delete and cancel the deletion if appropriate), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user the opportunity to confirm a deletion before performing the deletion but selecting content as the user is performing the selection gesture because selection is less intrusive than deletion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, after initiating a respective process of the process to delete the multiple text characters and the process to select the multiple text characters, and before detecting liftoff of the user input, the electronic device receives (952), via the touch-sensitive display, additional handwritten input, such as in
In some embodiments, in response to receiving the additional handwritten input, the electronic device continues (954) to perform the respective process based on the additional handwritten input independent of whether the additional handwritten input satisfies the one or more first criteria or the one or more second criteria, such as in
The above-described manner of selecting and deleting text (e.g., by performing a selection function or a deletion function if the handwritten input begins as a selection or deletion gesture, respectively) allows the electronic device to provide the user with certainty on the function that is performed (e.g., by committing to a particular function regardless of how the input gesture evolves from the initial gesture), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to begin the gesture and then still accepting further inputs to perform the initial function even if the further input deviates from the initial gesture), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, after initiating a respective process of the process to delete the multiple text characters and the process to select the multiple text characters, and before detecting liftoff of the user input, the electronic device receives (956), via the touch-sensitive display, additional handwritten input, such as in
In some embodiments, in response to receiving the additional handwritten input (958), in accordance with a determination that the additional handwritten input satisfies one or more first respective criteria, the electronic device performs (960) a selection process based on the handwritten input and the additional handwritten input, such as in
In some embodiments, in response to receiving the additional handwritten input (958), in accordance with a determination that the additional handwritten input satisfies one or more second respective criteria, the electronic device performs (962) a deletion process based on the handwritten input and the additional handwritten input, such as in FIG. 8HH (e.g., performing a deletion function over the entirety of the handwritten inputs (e.g., both the initial handwritten input and the additional handwritten input)). In some embodiments, the second criteria is satisfied if the additional handwritten input is a deletion gesture of a certain threshold (e.g., across a threshold number of characters (e.g., 3 characters, 5 characters, 1 word, 2 words, etc.) or for a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds)). In some embodiments, the second criteria is satisfied if the additional handwritten input causes the majority of the entirety of the handwritten input (e.g., the initial handwritten input and the additional handwritten input) to be a deletion gesture rather than a selection gesture (e.g., the additional handwritten input causes the majority of the entire handwritten input to be a deletion gesture or the additional handwritten input does not cause the majority of the handwritten to no longer be a deletion gesture).
The above-described manner of selecting and deleting text (e.g., by performing a selection function if the entirety of the handwritten satisfies a first criteria and performing a deletion function if the entirety of the handwritten input satisfies a second criteria) allows the electronic device to provide the user with the ability to change the function to be performed on-the-fly (e.g., by interpreting the handwritten input as a whole when determining whether the user is requesting to perform a deletion or selection option), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to begin with a particular gesture and switch to another gesture if the user changes his or her mind and performing the function that the user is requesting based on the user's gestures), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, the one or more first criteria are satisfied when the handwritten input strikes through the multiple text characters of the first editable text string along a direction of the first editable text string (964), such as in
In some embodiments, the one or more second criteria are satisfied when the handwritten input crosses out the multiple text characters of the first editable text string along a direction perpendicular to the direction of the first editable text string (966), such as in
The above-described manner of selecting and deleting text (e.g., by performing a selection function if the handwritten input strikes through multiple text characters and performing a deletion function if the handwritten input crosses through the multiple text characters vertically) allows the electronic device to provide the user with the ability to use the same input device to either select or delete text (e.g., by interpreting the handwritten input as selection or deletion based on the gesture performed by the handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by interpreting the handwritten input as a selection request or a deletion request based on the characteristics of the handwritten input, without requiring the user to navigate to a separate user interface to enable or disable selection or deletion functions), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, the one or more first criteria are satisfied when the handwritten input underlines the multiple text characters of the first editable text string (968), such as in
The above-described manner of selecting and deleting text (e.g., by performing a selection function if the handwritten input underlines multiple text characters and performing a deletion function if the handwritten input crosses through the multiple text characters vertically) allows the electronic device to provide the user with the ability to use the same input device to either select or delete text (e.g., by interpreting the handwritten input as selection or deletion based on the gesture performed by the handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by interpreting the handwritten input as a selection request or a deletion request based on the characteristics of the handwritten input, without requiring the user to navigate to a separate user interface to enable or disable selection or deletion functions), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, the handwritten input traverses the multiple text characters of the first editable text string (972), such as in
In some embodiments, the one or more second criteria are satisfied in accordance with a determination that the probability that the handwritten input corresponds to an input crossing out the multiple text characters is greater than the probability threshold (976), such as in
The above-described manner of selecting and deleting text (e.g., by performing a selection function if the handwritten input interacts with multiple text characters in a way that does not satisfy the deletion criteria and performing a deletion function if the handwritten input interacts with the text characters in a way that does satisfy the deletion criteria) allows the electronic device to provide the user with the ability to use the same input device to either select or delete text (e.g., by interpreting the handwritten input as selection unless the confidence that the handwritten input is a request to delete text is above a certain threshold level), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by defaulting to interpreting the handwritten input as a selection, without requiring the user to navigate to a separate user interface to enable or disable selection or deletion functions), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, the one or more first criteria are satisfied when the handwritten input comprises a double tap on the multiple text characters of the first editable text string (978), such as in
In some embodiments, the one or more second criteria are satisfied when the handwritten input crosses through two or more of the multiple text characters of the first editable text string (980), such as in
The above-described manner of selecting and deleting text (e.g., by performing a selection function if the handwritten input double taps on a word and performing a deletion function if the handwritten input crosses through the multiple text characters vertically) allows the electronic device to provide the user with the ability to use the same input device to either select or delete text (e.g., by interpreting the handwritten input as selection or deletion based on the gesture performed by the handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by interpreting the handwritten input as a selection request or a deletion request based on the characteristics of the handwritten input, without requiring the user to navigate to a separate user interface to enable or disable selection or deletion functions), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, the one or more first criteria are satisfied when the handwritten input moves in a closed (or substantially closed) shape that encloses at least a portion of the multiple text characters of the first editable text string (982), such as in
In some embodiments, the one or more second criteria are satisfied when the handwritten input crosses through two or more of the multiple text characters of the first editable text string (984), such as in
The above-described manner of selecting and deleting text (e.g., by performing a selection function if the handwritten input circles multiple text characters and performing a deletion function if the handwritten input crosses through the multiple text characters vertically) allows the electronic device to provide the user with the ability to use the same input device to either select or delete text (e.g., by interpreting the handwritten input as selection or deletion based on the gesture performed by the handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by interpreting the handwritten input as a selection request or a deletion request based on the characteristics of the handwritten input, without requiring the user to navigate to a separate user interface to enable or disable selection or deletion functions), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, while the multiple text characters in the first editable text string are selected (e.g., while the multiple text characters are highlighted), the device receives (986), via the touch-sensitive display, a user input comprising a handwritten input, such as in
In some embodiments, in response to receiving the user input (988), the device replaces (990) the multiple text characters in the first editable text string with respective editable text corresponding to the handwritten input, such as the replacement of the word “woke” with the word “got” in
In some embodiments, the handwritten input is converted to font-based text as described above with respect to methods 700, 1100, 1300, 1500, 1600, 1800, and/or 2000. In some embodiments, while receiving the handwritten input, the device displays a representation of the handwritten input (e.g., concurrently with the respective portion of the first editable text string) before converting the handwritten input to font-based text as described above with respect to methods 700, 1100, 1300, 1500, 1600, 1800, and/or 2000. In some embodiments, the respective portion of the first editable text string is replaced with font-based text corresponding to the handwritten input at the same time or after the handwritten input is converted to font-based text. In some embodiments, the newly inserted text is selected (e.g., highlighted). In some embodiments, the newly inserted text is not selected (e.g., not highlighted). In some embodiments, the characters immediately to the left and right of the replaced text is re-positioned to provide space for the newly inserted text (e.g., to provide the respective amount of character space). In some embodiments, if the handwritten input is not directed to the location corresponding to the respective portion of the first editable text string (e.g., does not satisfy the overlapping and/or threshold distance criteria), the electronic device does not replace the respective portion of the editable text string with font-based text corresponding to the handwritten input—in such embodiments, the electronic device optionally responds to the handwritten input such as described in methods 700, 1100, 1300, 1500, 1600, 1800, and/or 2000 (e.g., inserts the handwritten input at the respective location and converts to font-based text).
The above-described manner of replacing text (e.g., by receiving handwritten user input at or near selected text) provides a quick and efficient manner of replacing text using handwritten input, thus simplifying the interaction between the user and the electronic device and enhancing the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to select characters to be replaced and directly writing characters to replace the selected characters with the newly written characters without requiring the user to perform additional inputs to delete the undesired characters before inserting new characters), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
Users interact with electronic devices in many different manners, including entering text into the electronic device. In some embodiments, an electronic device displays text in a text field or a text region. The embodiments described below provide ways in which an electronic device inserts text into pre-existing text using a handwriting input device (e.g., a stylus). Enhancing interactions with a device reduces the amount of time needed by a user to perform operations, and thus reduces the power usage of the device and increases battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally referred to as a user of the device.
In
In some embodiments, in response to the user input, a space is created between the first and second portions of text, as shown in
In
In
In
In
In
In
In
In
In
In
In
In
In
In
In
In
For example, as shown in
FIGS. 10BBB-10III illustrate an embodiment of creating space between two characters. In FIG. 10BBB illustrates user interface 1000 in which text entry region 1002 includes one or more pre-existing text characters 1004. In some embodiments, the pre-existing text 1004 will be referred to as the first portion 1004-1 and second portion 1004-2, as shown in FIG. 10CCC, for ease of description. In FIG. 10CCC, a user input is detected from stylus 203 touching down in the space between first portion 1004-1 and second portion 1004-2. In FIG. 10DDD, the contact with the touch screen 504 is held for less than the threshold amount of time and no space is created between first portion 1004-1 and second portion 1004-2. In FIG. 10EEE, in response to the user maintaining contact with touch screen 504 for the threshold amount of time (e.g., 0.25 seconds, 0.5 seconds, 1 second, 3 seconds, 5 seconds, etc.), a space is created between first portion 1004-1 and second portion 1004-2 to provide the user with additional space to insert characters. In FIG. 10FFF, a termination of the user input (e.g., lift-off of contact with touch screen 504) is detected. In some embodiments, in response to detecting the termination of the user input, the space between first portion 1004-1 and second portion 1004-2 is maintained. In some embodiments, the space is maintained for a threshold amount of time (e.g., 0.25 seconds, 0.5 seconds, 1 second, 3 seconds, 5 seconds, 10 seconds, etc.) before the space is collapsed to the spacing from before the user input (e.g., as in FIG. 10BBB). It is understood that the above-described method of creating space between two characters is applicable to both font-based text and handwritten text (e.g., text that has not been converted into font-based text or text that was inserted using a drawing tool and will not be converted into font-based text but is still recognized as valid text).
In FIG. 10GGG, a user input is received from stylus 203 in text entry region 1002 writing the word “all” in the space created between first portion 1004-1 and second portion 1004-2. In some embodiments, while the user input is being received, a representation of the handwritten input 1006-1 is displayed at the location of the user input. In FIG. 10HHH, a termination of the user input (e.g., lift-off of contact with touch screen 504) is detected. In some embodiments, in response to detecting the termination of the user input, representation of the handwritten input 1006-1 is analyzed, valid characters are detected and converted into font-based text, as shown in FIG. 10III. In some embodiments, the detection and conversion of handwritten characters into font-based text is described with respect to methods 700, 900, 1300, 1500, 1600, 1800, and 2000. In some embodiments, after converting the handwritten input into font-based text (e.g., optionally after a threshold amount of time in which no input is received such as 1 second, 3 seconds, 5 seconds, 10 seconds, etc.) or in response to converting the handwritten input into font-based text, any additional space that is not occupied by the newly inserted characters is collapsed and the spacing between characters and words is reverted to their original setting, such as in FIG. 10III. Thus, in some embodiments, device 500 recognizes the handwritten input as valid characters and inserts the characters as font-based text (e.g., converts the handwritten input into font-based text and inserts the font-based text) into the respective line and/or sentence of text.
FIGS. 10JJJ-10MMM illustrate an embodiment of creating and removing space between two characters. In FIG. 10JJJ, a handwritten input is received from stylus 203 corresponding to a downward swipe gesture between the characters “no” and “where” of the word “nowhere” in pre-existing text 1004. In some embodiments, while receiving the handwritten input, a representation of the downward swipe 1030 is displayed in text entry region 1002. In some embodiments, a representation of the downward swipe 1030 is not displayed in text entry region 1002. In some embodiments, in response to the handwritten input, a whitespace character (e.g., a single space) is inserted between the characters “no” and “where” of the word “nowhere”, as shown in FIG. 10KKK. In some embodiments, a plurality of whitespace characters are inserted.
In FIG. 10LLL, a handwritten input is received from stylus 203 corresponding to a downward swipe gesture on the whitespace character between “no” and “where”. In some embodiments, while receiving the handwritten input, a representation of the downward swipe 1030 is displayed in text entry region 1002. In some embodiments, a representation of the downward swipe 1030 is not displayed in text entry region 1002. In some embodiments, in response to the handwritten input, the whitespace character between “no” and “where” is removed (e.g., resulting in the word “nowhere”), as shown in FIG. 10MMM. In some embodiments, device 500 removes only one whitespace character regardless of the number of whitespace characters between the two non-whitespace characters (e.g., if multiple whitespace characters exist). In some embodiments, device 500 removes all the whitespace characters between the two non-whitespace characters (e.g., if multiple whitespace characters exist). Thus, in some embodiments, a downward swipe gesture at a location between two adjacent non-whitespace characters causes insertion of a whitespace character whereas a downward swipe gesture at a location of a whitespace character causes the deletion of the whitespace character. In some embodiments, an upward swipe gesture also performs the insertion/deletion function described above. In some embodiments, the downward and/or upward swipe gesture need not be perfectly vertical. For example, a downward or upward swipe gesture that is 5 degrees off vertical, 10 degrees off vertical, 15 degrees off vertical, 30 degrees off vertical, etc. is recognizable as a request to insert or delete a whitespace character (as the case may be). It is understood that the above-described method of adding and removing whitespace characters between two characters is applicable to both font-based text and handwritten text (e.g., text that has not been converted into font-based text or text that was inserted using a drawing tool and will not be converted into font-based text but is still recognized as valid text).
FIGS. 10NNN-10SSS illustrate display of a text insertion indicator. In FIG. 10NNN, a user input is detected from stylus 203 touching down in the space between first portion 1004-1 and second portion 1004-2 of text in text entry region 1002 (e.g., similar to FIG. 10DDD). In FIG. 10OOO, the contact is maintained for the threshold amount of time (e.g., 0.25 seconds, 0.5 seconds, 1 second, 3 seconds, 5 seconds, etc.). In some embodiments, in response to the contact being maintained for the threshold amount of time, a space is created between first portion 1004-1 and second portion 1004-2 to provide the user with additional space to insert characters, and text insertion indicator 1032 is displayed at the location of the inserted space, as shown in FIG. 10OOO. As shown in FIG. 10OOO, text insertion indicator 1032 is displayed between first portion 1004-1 and second portion 1004-2 representing the space that was inserted for the user to provide additional handwritten input. In some embodiments, the height of text insertion indicator 1032 has a height taller than the height of the font-based text to provide enough height for handwritten input. In some embodiments, the height of text insertion indicator 1032 is the height of the font-based text (e.g., of pre-existing text characters 1004). As shown in FIG. 10OOO, text insertion indicator 1032 is a grey rectangle or a grey highlighting at the position of the inserted space.
In some embodiments, displaying text insertion indicator 1032 includes displaying an animation expanding text insertion indicator 1032 from an initial width (e.g., 0.5 character width, 1 character width, 2 character width, etc.) to the final width of the space that was inserted in conjunction with an animation of the movement of first portion 1004-1 to the left and/or the movement of second portion 1004-2 to the right. For example, in
In FIG. 10PPP, the animation of text insertion indicator 1032 continues and text insertion indicator 1032 further expands to reach its final width (e.g., the width of the space that was inserted). In some embodiments, second portion 1004-2 moves further rightwards to accommodate the entire width of the space that was inserted.
In FIG. 10QQQ, a termination of the user input (e.g., lift-off of contact with touch screen 504) is detected. In some embodiments, in response to detecting the termination of the user input, the space between first portion 1004-1 and second portion 1004-2 is maintained and display of text insertion indicator 1032 is maintained. In FIG. 10RRR, a handwritten input is received in the inserted space (e.g., at the location of text insertion indicator 1032). In some embodiments, while the user input is being received, a representation of the handwritten input 1006-1 is displayed at the location of the user input (e.g., within or on text insertion indicator 1032). In FIG. 10RRR, the handwritten input reaches the end of text insertion indicator 1032 (e.g., reaches the end of the inserted space, reaches within 0.5 mm, 1 mm, 3 mm, 5 mm, 1 cm, 3 cm, etc. of the end of text insertion indicator 1032). In some embodiments, in response to the handwritten input reaching the end of text insertion indicator 1032, additional space is inserted between first portion 1004-1 and second portion 1004-2 and text insertion indicator 1032 expands to include the width of the additional space, as shown in FIG. 10SSS. In some embodiments, second portion 1004-2 (or a portion of second portion 1004-2) is moved to a second line beneath first portion 1004-1 due to being displaced by the handwritten input.
In some embodiments, upon termination of the handwritten input and optionally after a threshold amount of time, representation of handwritten input 1006-1 is converted into font-based text (e.g., such as described above in FIG. 10III). In some embodiments, after the threshold amount of time, the spacing between the characters is collapsed to remove additional spaces that were not consumed by the additional handwritten input (e.g., such as described above in FIG. 10III). In some embodiments, concurrently with removing the additional spaces, text insertion indicator 1032 is ceased to be displayed (e.g., no longer displayed in user interface 1000).
It is understood that although the above examples describe and illustrate insertion of text between two words, inserting text between two characters in the same word or inserting text between any two characters based on the above-described exemplary methods are also possible.
As described below, the method 1100 provides ways to insert handwritten inputs into pre-existing text. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, an electronic device (e.g., an electronic device, a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device) including a touch screen, or a computer including a touch screen, such as device 100, device 300, device 500, device 501, or device 591) in communication with a touch-sensitive display displays (1102), on the touch-sensitive display, a text entry user interface including a first sequence of characters that includes a first portion of the first sequence of characters and a second portion of the first sequence of characters, such as in
In some embodiments, while displaying the text entry user interface, the electronic device receives (1104), via the touch-sensitive display, a user input in the text entry user interface in between the first portion of the first sequence of characters and the second portion of the first sequence of characters, such as in
In some embodiments, in response to receiving the user input (1106), in accordance with a determination that the user input corresponds to a request to enter respective font-based text in between the first portion of the first sequence of characters and the second portion of the first sequence of characters using handwritten input (e.g., a tap input with a stylus between two words or characters in a text string optionally indicates a request to enter text between the two words or character, respectively), the electronic device updates (1108) the text entry user interface by creating a space between the first portion of the first sequence of characters and the second portion of the first sequence of characters, wherein the space between the first portion and the second portion is configured to receive the handwritten input for inserting the respective font-based text between the first portion and the second portion of the first sequence of characters, such as in
In some embodiments, a touch-down of a stylus between two characters and continued contact for a threshold amount of time (e.g., 0.5 seconds, 1 second, 3 seconds, 5 seconds) indicates a request to enter text between the two characters. In some embodiments, an input with a particular pattern indicates a request to enter text between the two characters (e.g., a keyword gesture, or a keyword character, such as a caret). In some embodiments, beginning handwritten input with a stylus between the two characters (e.g., the user directly begins writing) indicates a request to enter text between the two words. In some embodiments, the system enters into a text insertion mode in response to the request to enter text between the first portion and the second portion of the first text string. In some embodiments, if the user input does not correspond to a request to enter font-based text, then interpret the input as a command or other non-text-entry gesture. For example, if the user input is optionally a request to scroll or navigate through the user interface (e.g., vertical or horizontal gestures), a selection input (e.g., a horizontal gesture passing through one or more characters), or a deletion input (e.g., a vertical cross-out gesture).
In some embodiments, the first portion of the text moves leftwards and the second portion of the text remains stationary. In some embodiments, the first portion of the text moves leftwards and the second portion of the text moves rightwards. In some embodiments, the first portion of the text remains stationary and the second portion of the text moves rightwards to create the space. In some embodiments, if the user has not entered handwritten input in the created space after a threshold amount of time (e.g., 1, 2, 5, 10 seconds), the first portion and second portion of the text are moved back together to form a continuous text string (e.g., back to its original state). In some embodiments, as the user enters handwritten input into the space, the space will increase in length (e.g., by continuing to push the first and/or second portions of the preexisting text string apart) to continually provide space for the user to continue inputting handwritten input. In some embodiments, after the user has stopped entering handwritten input for a threshold amount of time (e.g., 1, 2, 5, 10 seconds), the first portion and the second portion of the text will move to remove any excess space between the newly entered text and the preexisting text (e.g., the created excess space will collapse away). In some embodiments, the second portion of the text moves downwards (e.g., as opposed to rightwards) such that a new line is created (e.g., in response to the user reaching the end of the display or text field or in response to a user input corresponding to a request to insert a new line) to provide more space for the user to input handwritten input. In some embodiments, the handwritten input is converted into computer text as the user inputs the handwritten input (e.g., as described with reference to method 700). In some embodiments, the handwritten input is converted when the excess space is removed (e.g., when text insertion mode is terminated).
The above-described manner of inserting text (e.g., by receiving a user input corresponding to a request to insert text between pre-existing text and moving the pre-existing text to create space for the user to perform handwritten input) allows the electronic device to provide the user with the ability to insert handwritten input between preexisting text (e.g., by determining whether the user requests to insert text between pre-existing text and automatically moving the pre-existing text to create space for the user to insert handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to easily insert text between words without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text and to remove space after completion of text insertion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, after updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters, the electronic device receives (1110), via the touch-sensitive display, a handwritten input in the space between the first portion and the second portion of the first sequence of characters, such as in
In some embodiments, after receiving the handwritten input, the electronic device converts (1112) the handwritten input into font-based text in between the first portion and the second portion of the first sequence of characters, such as in
The above-described manner of inserting text (e.g., by receiving handwritten input in the space that was created and converting the handwritten input into font-based text and inserting the font-based text between the first portion and second portion of the sequence of) allows the electronic device to provide the user with the ability to insert handwritten input between preexisting text (e.g., by receiving handwritten text in the space that was created between the two portions of characters and inserting the font-based text that was converted from the handwritten text into that), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to easily insert text between words without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text and to remove space after completion of text insertion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the handwritten input is detected after detecting the user input in between the first portion and the second portion of the first sequence of characters without detecting lift-off from the touch-sensitive display (1114), such as in
The above-described manner of inserting text (e.g., by receiving handwritten input without detecting a lift-off of the input) allows the electronic device to provide the user with the ability to begin accepting handwritten input after creation of space between preexisting text (e.g., by accepting handwritten text in the space that was created between the two portions of characters without requiring or otherwise detecting a lift-off of the handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to begin handwritten input after the space has been created without lifting off from the screen), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the user input corresponds to the request to enter respective text in between the first portion and the second portion of the first sequence of characters using handwritten input when the user input comprises touchdown of a stylus on the touch-sensitive display in between the first portion and the second portion of the first sequence of characters, and updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters occurs in response to detecting the touchdown of the stylus before detecting further input from the stylus (1116), such as in
The above-described manner of inserting text (e.g., by beginning the process for inserting text upon detecting touchdown on the touch screen) allows the electronic device to provide the user with the ability to begin inserting handwritten text (e.g., by creating the space as soon as the user touches down on the screen, thus allowing the user to begin writing in the space that is created), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to easily insert text by merely touching down on the desired location and without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the touchdown of the stylus is between two words of the first sequence of characters (1118), such as in
The above-described manner of inserting text (e.g., by receiving a request to insert text between two words) allows the electronic device to provide the user with the ability to insert handwritten input between preexisting text (e.g., by receiving a touchdown between two words and allowing insertion of text between the two words), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to easily insert text between words without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text and to remove space after completion of text insertion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the user input corresponds to the request to enter respective text in between the first portion and the second portion of the first sequence of characters using handwritten input when the user input comprises touchdown of a stylus on the touch-sensitive display for longer than a time threshold (e.g., 1, 2, 3, 5 seconds). In some embodiments, the input corresponding to the request to insert text is a long touch by the stylus on the touch screen), and updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters occurs in response to detecting the touchdown of a stylus on the touch-sensitive display for longer than the time threshold (1120), such as in
In some embodiments, the input is also required to be substantially stationary for the time threshold (e.g., no more than a threshold amount of movement of the stylus during the time threshold). In some embodiments, entering into insertion mode after a long hold allows the system to determine that the user did not inadvertently request insertion of text. In some embodiments, if the touchdown is not longer than the time threshold, then the user input is ignored or otherwise not interpreted as a request to enter respective text. In some embodiments, the user input that is not longer than the time threshold is interpreted as a selection input. In some embodiments, the user input that is not longer than the tine threshold causes a pop-up or other menu to be displayed to allow the user to determine what function to perform.
The above-described manner of inserting text (e.g., by interpreting a long press user input as a request to insert text between pre-existing text and moving the pre-existing text to create space for the user to perform handwritten input) allows the electronic device to provide the user with the ability to insert handwritten input between preexisting text (e.g., by interpreting a long press user input as a request to insert text between pre-existing text and automatically moving the pre-existing text to create space for the user to insert handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by ensuring that the user is requesting to insert text by interpreting a long press input as a request to insert text without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text and to remove space after completion of text insertion), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the user input corresponds to the request to enter respective text in between the first portion and the second portion of the first sequence of characters using handwritten input when the user input comprises a respective gesture (e.g., receiving a particular keyword gesture that indicates a request to insert text), and updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters occurs in response to detecting the respective gesture (1122), such as in
In some embodiments, receiving a caret gesture between two portions of sequence of characters is considered a request to insert text between the two portions of sequence of characters. In some embodiments, if the user input does not comprise a respective gesture (e.g., the user input is another gesture that is not considered a keyword gesture for inserting text), then the user input is not interpreted as a request to insert text. In some embodiments, the user input that does not comprise a respective gesture is interpreted as a selection input, a deletion input, or a navigation input, etc.
The above-described manner of inserting text (e.g., by interpreting a handwritten input of a particular respective gesture as a request to insert text between pre-existing text and moving the pre-existing text to create space for the user to perform handwritten input) allows the electronic device to provide the user with the ability to insert handwritten input between preexisting text (e.g., by interpreting a respective gesture in the handwritten input as a request to insert text between pre-existing text and automatically moving the pre-existing text to create space for the user to insert handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to easily insert text between words without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the user input comprises touchdown of a stylus on the touch-sensitive display (1124), such as in
In some embodiments, while displaying the selectable option for creating the space between the first and second portions of the first sequence of characters, the electronic device receives (1128), via the touch-sensitive display, selection of the selectable option, such as in
In some embodiments, updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters occurs in response to detecting the selection of the selectable option (1130), such as in
The above-described manner of inserting text (e.g., by receiving a user selection of a selectable option for inserting text and moving the pre-existing text to create space for the user to perform handwritten input) allows the electronic device to provide the user with the ability to insert handwritten input between preexisting text (e.g., by displaying a menu including selectable option to insert text and automatically moving the pre-existing text to create space for the user to insert handwritten input in response to the user's selection of the selectable option), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to insert text between words by selecting a selectable option to insert text without requiring the user to navigate to a separate user interface or menu to create space to insert text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, after updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters, the electronic device receives (1132), via the touch-sensitive display, a handwritten input in the space between the first portion and the second portion of the first sequence of characters, such as in
In some embodiments, in response to receiving the handwritten input (1134), the electronic device displays (1136) a representation of the handwritten input in the space between the first and second portions of the first sequence of characters, such as in
In some embodiments, in response to receiving the handwritten input (1134), in accordance with a determination that the handwritten input satisfies one or more criteria (e.g., reaches near the end of the space, includes special gesture to add more space, etc.), the electronic device expands (1138) the space between the first and second portions of the first sequence of characters, such as in
The above-described manner of further providing space for inserting text (e.g., by receiving handwritten input directed to the space created between the first and second portions of text and further moving the first and/or second portions of text to create more space for the user to continue handwritten input as the user continues to provide handwritten input) allows the electronic device to provide the user with the ability to continue inserting handwritten input between preexisting text (e.g., by continuing to move the pre-existing text to continue to provide space for the user to input handwritten inputs), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to easily continue inserting text even after exhausting the initial space created for inserting text without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the handwritten input satisfies the one or more criteria when the handwritten input includes a first respective gesture, and does not satisfy the one or more criteria when the handwritten input includes a second respective gesture, different than the first respective gesture (1140), such as in
The above-described manner of further providing space for inserting text (e.g., by receiving handwritten input with a particular keyword gesture and further moving the first and/or second portions of text to create more space for the user to continue handwritten input as the user continues to provide handwritten input) allows the electronic device to provide the user with the ability to continue inserting handwritten input between preexisting text (e.g., by moving the pre-existing text to provide further space for the user to input handwritten inputs in response to receiving a particular keyword gesture), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to easily continue inserting text even after exhausting the initial space created for inserting text without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, after updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters, the electronic device receives (1142), via the touch-sensitive display, a handwritten input in the space between the first portion and the second portion of the first sequence of characters, such as in
In some embodiments, in response to receiving the handwritten input (1144), the electronic device displays (1146) a representation of the handwritten input in the space between the first and second portions of the first sequence of characters, such as in
In some embodiments, in response to receiving the handwritten input (1144), in accordance with a determination that one or more new line criteria are satisfied, the electronic device updates (1148) the user interface to create a new line configured to receive additional handwritten input for inserting additional respective text in the new line, such as in
The above-described manner of inserting a new line for further inserting text (e.g., by receiving handwritten input and inserting a new line in the pre-existing text if the new line criteria are satisfied) allows the electronic device to provide the user with the ability to insert multi-lined text (e.g., by automatically determining whether a new line should be inserted and inserting the new line to provide space for the user to further input handwritten inputs), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to easily insert a new line in the pre-existing text without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more new line criteria include a criterion that is satisfied when the handwritten input reaches an end of a current line in the user interface (1150), such as in
The above-described manner of inserting a new line for further inserting text (e.g., by receiving handwritten input and inserting a new line in the pre-existing text if the handwritten input reaches the end or near the end of the current line of text) allows the electronic device to provide the user with the ability to insert multi-lined text (e.g., by automatically determining that a user likely needs a new line to further enter handwritten text and inserting the new line to provide space for the user to further input handwritten inputs), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically inserting a new line in a situation in which a new line is likely needed without requiring the user to navigate to a separate user interface or menu or perform additional user inputs to create space to insert text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more new line criteria include a criterion that is satisfied when the additional handwritten input is detected below existing font-based text in the user interface (1152), such as in
The above-described manner of inserting a new line for further inserting text (e.g., by receiving handwritten input that is below the existing line of text and inserting a new line at the location below the existing line of text) allows the electronic device to provide the user with the ability to insert multi-lined text (e.g., by automatically interpreting the handwritten input below the existing font-based text as a request to insert a new line at the location of the handwritten input and inserting the new line to provide space for the user to further input handwritten inputs), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically inserting a new line when the user provides handwritten input below the existing font-based text indicating a request to insert a new line at the location of the handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more new line criteria include a criterion that is satisfied when a tap input is detected below existing font-based text in the user interface (1154), such as in
The above-described manner of inserting a new line for further inserting text (e.g., by receiving a tap input below the existing line of text and inserting a new line at the location below the existing line of text) allows the electronic device to provide the user with the ability to insert multi-lined text (e.g., by interpreting a tap input below the existing font-based text as a request to insert a new line at the location of the handwritten input and inserting the new line to provide space for the user to further input handwritten inputs), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by inserting a new line when the user taps at a location below existing font-based text indicating a request to insert a new line at the location of the handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in response to receiving the handwritten input (1156), in accordance with a determination that the handwritten input is within a threshold distance of an end of a current line in the user interface, the electronic device displays (1158), in the user interface, a selectable option for creating a new line in the user interface, such as in
In some embodiments, the one or more new line criteria include a criterion that is satisfied when selection of the selectable option for creating the new line in the user interface is detected (1160), such as in
The above-described manner of inserting a new line for further inserting text (e.g., by displaying a selectable option that is selectable to insert a new line and inserting a new line in response to receiving a user input selecting the selectable option for inserting a new line) allows the electronic device to provide the user with the ability to insert multi-lined text (e.g., by dynamically displaying a selectable option to insert a new line when the user's handwriting input reaches the end of a line and a new line is likely needed, and inserting a new line in response to receiving a user input selecting the selectable option), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by inserting a new line when the user selects a selectable option for inserting a new line that is displayed when the user reaches the end of the current line), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while the new line configured to receive the additional handwritten input is included in the user interface, the electronic device receives (1162), via the touch-sensitive display, a respective user input, such as in
In some embodiments, in response to receiving the respective user input (1164), in accordance with a determination that the respective user input comprises a tap input detected at an end of a last word in a previous line, previous to the new line, in the user interface, or a tap input detected at a beginning of a first word in the new line in the user interface, the electronic device displays (1166), in the user interface, a selectable option for removing the new line from the user interface, such as in
The above-described manner of removing a line break in multi-lined text (e.g., by receiving an input at the end of a first line or the beginning of a second line, displaying a selectable option for removing the line break between the first line and the second line, and removing the line break in response to receiving a user input selecting the selectable option) allows the electronic device to provide the user with the ability to remove a line break in multi-lined text (e.g., by dynamically displaying a selectable option to remove a line break and removing the line break in response to the user's selection of the selectable option to remove the line break), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with a selectable option to remove a line break and removing the line break in response to receiving a user input selecting the selectable option), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while the new line configured to receive the additional handwritten input is included in the user interface and the new line includes a respective sequence of characters, the electronic device receives (1168), via the touch-sensitive display, a respective input including a touchdown of a stylus on the respective sequence of characters and a movement of the stylus to a respective line, different than the new line, in the user interface, such as in
In some embodiments, in response to receiving the respective input (1170), the electronic device moves (1172) the respective sequence of characters to the respective line in the user interface, such as in
In some embodiments, in response to receiving the respective input (1170), the electronic device removes (1174) the new line from the user interface, such as in
The above-described manner of removing a line break in multi-lined text (e.g., by receiving an input at a second line of text that drags the second line of text to a first line of text and removing any line breaks between the first and second lines of text) allows the electronic device to provide the user with the ability to remove a line break in multi-lined text (e.g., by interpreting the user's gesture dragging a line to a previous line as a request to remove a line break between the two lines of and removing the line break in response to the user's request to remove the line break), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with an intuitive method of moving text and automatically removing line breaks in accordance with the user's inputs without requiring the user to navigate to a separate user interface or perform additional inputs to remove line breaks), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, after updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters, the electronic device receives (1176), via the touch-sensitive display, a handwritten input in the space between the first portion and the second portion of the first sequence of characters, such as in
In some embodiments, in response to receiving the handwritten input (1178), the electronic device displays (1180), in the user interface, a representation of the handwritten input in the space between the first and second portions of the first sequence of characters, such as in
In some embodiments, in response to receiving the handwritten input (1178), in accordance with a determination that the handwritten input has not reached an end of a current line in the user interface, the electronic device ceases (1182) to display the representation of the handwritten input after a first elapsed time since receiving the handwritten input, such as in FIG. 10AAA (e.g., begin converting the handwritten text into font-based text). In some embodiments, the conversion is performed after a certain time delay. In some embodiments, the conversion is performed according to method 700 and/or method 1300. In some embodiments, if the progress of the handwritten input is at a position before a certain threshold location (e.g., before reaching the halfway point, before reaching the ¾ point, then convert the text according to the ordinary timing of converting text).
In some embodiments, in response to receiving the handwritten input (1178), in accordance with a determination that the handwritten input has reached the end of the current line in the user interface, the electronic device ceases (1184) to display the representation of the handwritten input after a second elapsed time, shorter than the first elapsed time, since receiving the handwritten input, such as in
The above-described manner of providing space for handwritten input (e.g., by converting text at a faster speed as the user begins to run out of space to provide handwritten) allows the electronic device to continuously provide the user with space to input handwritten inputs (e.g., by determining that the user will run out of space for handwritten input and increasing the speed of converting handwritten text into font-based text in order to remove the handwritten text from display to free up space for the user to continue providing handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically and continuously providing space for the user to input handwritten text by converting previously written handwritten text at a faster speed without requiring the user to wait for the conversion process to occur or perform additional inputs to create space for further handwritten text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, after updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters, the electronic device receives (1186), via the touch-sensitive display, a handwritten input in the space between the first portion and the second portion of the first sequence of characters, such as in
In some embodiments, after receiving the handwritten input (1188), in accordance with a determination that no additional handwritten input is received for a time threshold after an end of the handwritten input, the electronic device reduces (1190) a size of the space between the first portion and the second portion of the first sequence of characters to remove space not consumed by the handwritten input in the user interface, such as in
The above-described manner of removing excess space after handwritten input (e.g., by removing excess space between the text that was created to make space for the handwritten input after handwritten input has ceased for a threshold amount of time) allows the electronic device to exit text insertion mode (e.g., by determining that the user has stopped inserting text and removing any excess space to align the inserted text with the pre-existing text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically exiting text insertion mode and removing excess space without requiring the user to perform additional inputs to remove excess space after inserting handwritten inputs), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, after updating the text entry user interface by creating the space between the first portion of the first sequence of characters and the second portion of the first sequence of characters, the electronic device receives (1192), via the touch-sensitive display, a handwritten input in the space between the first portion and the second portion of the first sequence of characters, such as in
In some embodiments, after receiving the handwritten input (1194), in accordance with a determination that no additional handwritten input is received for a time threshold after an end of the handwritten input (e.g., 1 second, 2 seconds, 3 seconds, 5 seconds, etc.), the electronic device converts (1196) the handwritten input into font-based text in the space between the first and second portions of the first sequence of characters, such as in
The above-described manner of inserting handwritten input (e.g., by converting the handwritten input after the user has ceased input for a threshold amount of time) allows the electronic device to insert text (e.g., by converting the handwritten input and insert the converted text into the space between the first and second portions of text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically converting handwritten input into font-based text and inserting the font-based text between the first and second portions of text when it appears that the user has completed handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the electronic device displays (1198), in the text entry user interface, a second sequence of characters that includes a first portion of the second sequence of characters and a second portion of the second sequence of characters, such as in
In some embodiments, while displaying the text entry user interface, the electronic device receives (1198-2), via the touch-sensitive display, a second user input in the text entry user interface in between the first portion of the second sequence of characters and the second portion of the second sequence of characters, such as in
In some embodiments, in response to receiving the second user input (1198-4), in accordance with a determination that the second user input corresponds to a request to enter second respective font-based text in between the first portion of the second sequence of characters and the second portion of the second sequence of characters using handwritten input (1198-6), the electronic device displays (1198-8), in the user interface, a handwritten input user interface element (e.g., overlaid on what was previously displayed in the user interface) configured to receive handwritten input for inserting the second respective font-based text between the first portion and the second portion of the second sequence of characters, such as in
The above-described manner of inserting handwritten input (e.g., by displaying a pop-up user interface element with a text box in which the user inserts handwritten input for conversion and insertion into the pre-existing text) allows the electronic device to provide the user with a text insertion element (e.g., by displaying a text box in response to the user's request to insert text, accepting handwritten input in the text box, and converting the handwritten input into font-based text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying a text insertion user interface element in which the user is able to input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the handwritten input user interface element, the electronic device receives (1198-10), via the touch-sensitive display, a second handwritten input in the handwritten input user interface element, such as in
In some embodiments, in response to receiving the second handwritten input in the handwritten input user interface element (1198-12), the electronic device inserts (1198-14) font-based text corresponding to the second handwritten input into the text entry user interface, such as in
In some embodiments, in response to receiving the second handwritten input in the handwritten input user interface element (1198-12), while the handwritten input user interface element remains stationary on the touch-sensitive display, the electronic device scrolls (1198-16) the text entry user interface in accordance with movement of a current text insertion point, such as in
The above-described manner of inserting handwritten input (e.g., by scrolling the user interface behind the pop-up text box as the user continue to input handwritten input) allows the electronic device to provide the user with a stationary text insertion element (e.g., by maintaining the location of the pop-up text box and scrolling the user interface behind the pop-up text box when needed to maintain display of the insertion point), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by maintaining the location of the pop-up text box while simultaneously displaying the insertion point without requiring the user to readjust his or her handwriting position while providing handwriting inputs), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the handwritten input user interface element, the electronic device receives (1198-18), via the touch-sensitive display, a second handwritten input in the handwritten input user interface element, such as in
In some embodiments, in response to receiving the second handwritten input in the handwritten input user interface element (1198-20), the electronic device displays (1198-22), in the handwritten input user interface element, a representation of the second handwritten input, such as in
In some embodiments, in response to receiving the second handwritten input in the handwritten input user interface element (1198-20), in accordance with a determination that the second handwritten input has not reached an end of the handwritten input user interface element, the electronic device ceases (1198-24) to display the representation of the second handwritten input after a first elapsed time since receiving the second handwritten input, such as in FIG. 10AAA (e.g., begin converting the handwritten text into font-based text. In some embodiments, the conversion is performed after a certain time delay). In some embodiments, the conversion is performed according to method 700 and/or method 1300. In some embodiments, if the progress of the handwritten input is at a position before a certain threshold location (e.g., before reaching the halfway point, before reaching the ¾ point, then convert the text according to the ordinary timing of converting text.
In some embodiments, in response to receiving the second handwritten input in the handwritten input user interface element (1198-20), in accordance with a determination that the second handwritten input has reached the end of the handwritten input user interface element, the electronic device ceases (1198-26) to display the representation of the second handwritten input after a second elapsed time, shorter than the first elapsed time, since receiving the second handwritten input, such as in
The above-described manner of providing space for handwritten input (e.g., by converting text at a faster speed as the user begins to run out of space to provide handwritten) allows the electronic device to continuously provide the user with space to input handwritten inputs (e.g., by determining that the user will run out of space for handwritten input and increasing the speed of converting handwritten text into font-based text in order to remove the handwritten text from display to free up space for the user to continue providing handwritten input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically and continuously providing space for the user to input handwritten text by converting previously written handwritten text at a faster speed without requiring the user to wait for the conversion process to occur or perform additional inputs to create space for further handwritten text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the text entry user interface including the first sequence of characters, the device receives (1198-28), via the touch-sensitive display, a respective user input including a movement across a respective portion of the first sequence of characters (e.g., a downward or an upward movement across the respective portion of first sequence of characters) while maintaining contact with the touch-sensitive display at a location between a first character and a second character in the first sequence of characters, such as in FIGS. 10JJJ and 10LLL (e.g., a vertical or downward or upward swipe gesture between two characters (optionally adjacent characters).
In some embodiments, the first sequence of characters is a sequence of handwritten characters. In some embodiments, the first sequence of characters is font-based text. In some embodiments, the first sequence of characters is includes some font-based text and some handwritten characters. In some embodiments, the downward swipe gesture is less than a threshold angle from vertical (e.g., 5 degrees from vertical, 10 degrees from vertical, 20 degrees from vertical, etc.) and need not be perfectly vertical. In some embodiments, the input is from a stylus or similar input device in contact with the touch-sensitive display.
In some embodiments, in response to receiving the respective user input (1198-30), in accordance with a determination that no characters separate the first character and the second character in the first sequence of characters (e.g., the first character and second character are adjacent characters without a whitespace character (e.g., space) between them), the device updates (1198-32) the text entry user interface by adding a whitespace character between the first character and the second character in the first sequence of characters, such as in FIG. 10KKK (e.g., automatically inserting a whitespace character (e.g., single space) between the first and second characters). In some embodiments, a plurality of whitespace characters are inserted.
In some embodiments, in accordance with a determination that only a whitespace character separates the first character and the second character in the first sequence of characters, the device updates (1198-34) the text entry user interface by removing the whitespace character between the first character and the second character in the first sequence of characters, such as in FIG. 10MMM (e.g., if the first and second characters are separated by a single whitespace character, and no other characters, then remove the whitespace character, thus making the two characters adjacent).
In some embodiments, if the first and second characters are separated by multiple whitespace characters, then remove a single whitespace character. In some embodiments, if the first and second characters are separated by multiple whitespace characters, then remove all the whitespace characters between the first and second characters, thus making the two characters adjacent.
The above-described manner of inserting and removing whitespace (e.g., by receiving a downward swipe between two text characters) provide the user with a quick and efficient method of separating or adjoining characters (e.g., by automatically adding whitespace if no whitespace exists and removing whitespace if whitespace already exists), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by performing both an addition and deletion function using the same gesture without requiring the user to perform additional inputs or different inputs to either add or remove whitespace), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
Users interact with electronic devices in many different manners, including entering text into the electronic device. In some embodiments, an electronic device receives handwritten input from a handwriting input device (e.g., a stylus) and converts the handwritten input into font-based text (e.g., computer text, digital text, etc.). The embodiments described below provide ways in which an electronic device manages the timing of converting handwritten input from a handwriting input device (e.g., a stylus) into font-based text (e.g., computer text, digital text, etc.). Enhancing interactions with a device reduces the amount of time needed by a user to perform operations, and thus reduces the power usage of the device and increases battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally referred to as a user of the device.
In some embodiments, as shown in
In
In
In
In
In
In some embodiments, handwritten input 1204-3 is converted into font-based text when stylus 203 is determined to be a threshold distance away from device 500 (e.g., 6 inches, 1 foot, 2 feet, outside of wireless communication range, etc.). In some embodiments, handwritten input 1204-3 is converted into font-based text when stylus 203 is determined to be pointed away from device 500 (e.g., the tip or the writing end of stylus 203 is facing away from device 500). In some embodiments, handwritten input 1204-3 is converted into font-based text when stylus 203 is docked with device 500 (e.g., magnetically attached to device 500, being charged by device 500, or otherwise in a state of non-use). Thus, based on the context of stylus 203 itself (e.g., location, distance, angle, movement, or any other indication that the user is done using the stylus for handwritten input, etc.), handwritten inputs are optionally converted into font-based text.
In
In some embodiments, different predetermined delay times are used for converting handwritten input into font-based text based on the context and the handwritten input conversion mode of the device. In some embodiments, when device 500 is in a live conversion mode (e.g., a mode in which letters or words are converted while the user is still performing handwritten inputs), a shorter predetermined delay time (e.g., 0.5 seconds, 1 second, 2 seconds, 5 seconds) is used when certain criteria for faster conversion times are satisfied, as will be discussed in further detail below. In some embodiments, when device 500 is in a live conversion mode, a longer predetermined delay time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds, 10 seconds) is used when certain criteria for slower conversion times are satisfied, as will be discussed in further detail below. While in live conversion mode, in some embodiments, each letter or word has its own respective timer for controlling the timing for converting the respective letter or word into font-based text. In some embodiments, a third, even longer predetermined delay time is used when device 500 is in a simultaneous conversion mode (e.g., a mode in which an entire sequence of letters or words are converted at one time after the user has completed the sequence of handwritten inputs). In simultaneous conversion mode, in some embodiments, the entire sequence of letters or words has a timer for controlling the timing for converting the sequence of letters or words into font-based text.
In
In
In
In
In
In
In
As described below, the method 1300 provides ways to manage the timing of converting handwritten text into font-based text. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, an electronic device (e.g., an electronic device, a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device) including a touch screen, or a computer including a touch screen, such as device 100, device 300, device 500, device 501, or device 591) in communication with a touch-sensitive display displays (1302), on the touch-sensitive display, a text entry user interface, such as in
In some embodiments, while displaying the text entry user interface, the electronic device receives (1304), via the touch-sensitive display, a first sequence of one or more handwritten user inputs in the text entry user interface, such as in
In some embodiments, while receiving the first sequence of one or more handwritten user inputs, the electronic device displays (1306), on the touch-sensitive display, a visual representation of the first sequence of one or more handwritten user inputs in the text entry user interface, such as in
In some embodiments, in response to detecting an end of the first sequence of one or more handwritten user inputs (1308) (e.g., any suitable termination of the sequence of handwritten user inputs), in accordance with a determination that a context associated with the first sequence of one or more handwritten user inputs satisfies one or more first criteria (e.g., text conversion criteria for converting handwritten input into font-based text without waiting for other predetermined conditions, for example), the electronic device replaces (1310) the visual representation of the first sequence of one or more handwritten user inputs with text corresponding to the first sequence of one or more handwritten user inputs without regard to whether or not respective timing criteria have been met, such as in
For example, if the user stops performing handwritten input (e.g., for a threshold amount of time, such as 1, 3 or 5 seconds), then the sequence of handwritten inputs is considered to have ended. In some embodiments, if the user completes writing a character, a word, or a sentence, the sequence of handwritten inputs is considered to have ended. In some embodiments, the handwritten input does not necessarily need to complete writing a sentence, a word, or a character, to be considered an end of the handwritten input. For example, if the user stops inputting mid-sentence, mid-word, or mid-character, the sequence of handwritten inputs is optionally considered terminated. In some embodiments, if another user input is detected while receiving handwritten input (e.g. or optionally between receiving handwritten words, characters, or sentences), the sequence of handwritten inputs is considered terminated.
For example, a triggering event optionally causes the handwritten input to be converted to computer text at that time, without waiting for other predetermined conditions to be met (e.g., without regard to any timers). In some embodiments, if a user enters handwritten input in one text field and selects another text field, the handwritten input in the first text field is converted to computer text. In some embodiments, if the user enters handwritten input and then interacts with another user interface element or scrolls the user interface, the handwritten input is converted to computer text. In some embodiments, if the user enters handwritten input using the stylus and subsequently interacts with the screen using a finger, the handwritten input is converted to computer text. In some embodiments, if the user enters handwritten input using the stylus and places the stylus down, moves the stylus away from the touch screen, or puts the stylus away (e.g., based on measurements from an accelerometer, gyroscope, or other positional and/or rotational sensing mechanism in the stylus), the handwritten input is converted to computer text.
In some embodiments, in response to detecting an end of the first sequence of one or more handwritten user inputs (1308) (e.g., any suitable termination of the sequence of handwritten user inputs), in accordance with a determination that the context associated with the first sequence of one or more handwriting user inputs does not satisfy the one or more first criteria, the electronic device delays (1312) replacing the visual representation of the first sequence of one or more handwriting user inputs with the text corresponding to the first sequence of one or more handwriting user inputs until the respective timing criteria have been met, such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to text under certain conditions and by delaying conversion for a certain amount of time under other conditions) allows the electronic device to convert text when it appears that the user has completed handwritten input (e.g., by converting the text in certain situations that indicates that the user has finished writing, and by not converting (or delaying the conversion) when it does not appear as if the user has completed writing), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying to the user the results of his handwriting input as soon as possible (e.g., in situations in which it appears that the user has completed writing) without unduly distracting the user when the user appears to still be writing, without requiring the user to always wait for conversion even when the user has completed writing or to have text converted prematurely before the user has finished writing), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the one or more first criteria are satisfied when the first sequence of one or more handwritten user inputs includes more than a threshold number of words followed by a space (1314), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after the user has written a threshold number of words) allows the electronic device to convert text after the user has written a certain number of words (e.g., by converting the text in a situation in which converting the word would not distract the user's handwriting input and balances the time delay before words are converted into font-based text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying to the user the results of his or her handwriting input as soon as possible while without unduly distracting the user when the user is still be writing, without requiring the user to wait for conversion even when the user has completed writing or to have text converted prematurely before the user has finished writing a word or sentence), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the one or more first criteria are satisfied when the first sequence of one or more handwritten user inputs is directed to a first text entry region in the text entry user interface, and the end of the first sequence of one or more handwritten user inputs includes input directed to a second text entry region in the text entry user interface (1316), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after the user indicates a request to insert text in another text entry region) allows the electronic device to convert text after the user has completed handwritten input in a text entry region (e.g., by converting the text when the user signals that the user is completed entering handwritten text in the text entry region by selecting another text entry region to enter text into), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying to the user the results of his or her handwriting input as soon as possible when the user appears to be finished inputting handwritten inputs in the first text entry region, without requiring the user to wait for conversion even when the user has completed writing), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the text entry user interface includes a selectable option for performing an action, and the one or more first criteria are satisfied when the end of the first sequence of one or more handwritten user inputs includes selection of the selectable option (1318), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after the user interacts with another user interface including selecting a selectable option) allows the electronic device to convert text after the user has completed handwritten input in a text entry region (e.g., by converting the text when the user signals that the user is completed entering handwritten text in the text entry region by selecting a selectable option), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying to the user the results of his or her handwriting input as soon as possible when the user appears to be finished inputting handwritten inputs in the first text entry region, without requiring the user to wait for conversion even when the user has completed writing), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the first sequence of one or more handwritten user inputs comprise stylus input detected on the touch-sensitive display, and the one or more first criteria are satisfied when an input comprising a finger input is detected on the touch-sensitive display (1320), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after the user interacts with the touch screen with a finger) allows the electronic device to convert text after the user has completed handwritten input in a text entry region (e.g., by converting the text when the user signals that the user is completed entering handwritten text in the text entry region by switching to using a finger instead of the stylus), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying to the user the results of his or her handwriting input as soon as possible when the user appears to be finished inputting handwritten inputs in the first text entry region, without requiring the user to wait for conversion even when the user has completed writing), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the one or more first criteria are satisfied when a scrolling input is detected on the touch-sensitive display (1322), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after the user performs a scrolling input) allows the electronic device to convert text after the user has completed handwritten input in a text entry region (e.g., by converting the text when the user signals that the user is completed entering handwritten text in the text entry region by performing a scrolling input), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying to the user the results of his or her handwriting input as soon as possible when the user appears to be finished inputting handwritten inputs in the first text entry region, without requiring the user to wait for conversion even when the user has completed writing), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the first sequence of one or more handwritten user inputs comprise stylus input detected on the touch-sensitive display, and the one or more first criteria are satisfied in accordance with a determination that the stylus has been placed down on a surface by a user (1324), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after the user places the stylus down) allows the electronic device to convert text after the user has completed handwritten input in a text entry region (e.g., by converting the text when the user signals that the user is completed entering handwritten text in the text entry region by placing the stylus down), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying to the user the results of his or her handwriting input as soon as possible when the user appears to be finished inputting handwritten inputs in the first text entry region, without requiring the user to wait for conversion even when the user has completed writing), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the first sequence of one or more handwritten user inputs comprise stylus input detected on the touch-sensitive display, and the one or more first criteria are satisfied when the stylus has moved more than a threshold distance (e.g., 0.5 cm, 1 cm, 3 cm, 5 cm) from the touch-sensitive display (1326), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after the user moves the stylus a threshold distance away from the touch screen) allows the electronic device to convert text after the user has completed or is pausing handwritten input in a text entry region (e.g., by converting the text when the user signals that the user is completed entering handwritten text in the text entry region or has paused handwritten input in the text entry region by moving the stylus a threshold distance away from the touch screen), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying to the user the results of his or her handwriting input as soon as possible when the user appears to be finished or appears to have paused inputting handwritten inputs in the first text entry region, without requiring the user to wait for conversion even when the user has completed writing), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, in accordance with a determination that one or more second criteria have been satisfied, the respective timing criteria have been met when a first time threshold has elapsed since the end of the first sequence of one or more handwritten user inputs (1328), such as in
In some embodiments, in accordance with a determination that one or more third criteria have been satisfied, the respective timing criteria have been met when a second time threshold, longer than the first time threshold, has elapsed since the end of the first sequence of one or more handwritten user inputs (1330), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after a predetermined amount of timer based on the context of the handwritten input) allows the electronic device to convert text after the user has likely completed writing a word or at a point that is least intrusive (e.g., by using a shorter timer to convert text in certain situations when the user has likely completed writing a word or sentence and by using a longer timer to convert text in situations when a user potentially could input further letters or words), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by converting handwritten input at a time when it is least intrusive while providing the user the opportunity to continue writing even if the user has momentarily paused writing), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the one or more second criteria have been satisfied when the end of the first sequence of one or more handwritten user inputs comprises a request to add punctuation to the sequence of characters (1332), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after a shorter timer delay after detecting that the user has input punctuation) allows the electronic device to convert text after the user has likely completed writing a word or at a point that is least intrusive (e.g., by using a shorter timer to convert text when the user has input a punctuation and it is likely that the user has completed writing a word or sentence), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by converting handwritten input at a time when it is least intrusive and likely to have completed writing a word or sentence), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the one or more second criteria have been satisfied when the one or more handwritten user inputs ends with a word to which a character cannot be added (1334), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after detecting a word in which no further characters can be added) allows the electronic device to convert text after the user has likely completed writing a word (e.g., by using a shorter timer to convert text when the user has input a word in which no further letters can be added and it is likely that the user has completed writing the word), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by converting handwritten input at a time when it is least intrusive and likely to have completed writing a word), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the one or more third criteria have been satisfied when the end of the first sequence of one or more handwritten user inputs comprises a pause for longer than a time threshold (1336), such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting to font-based text after a longer timer if no other criteria for faster conversion is satisfied) allows the electronic device to convert text after a certain time delay (e.g., by using a longer timer to convert text when none of the other faster conversion situations apply), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by ensuring that handwritten input is converted without too much delay without requiring the user to perform additional inputs to cause the conversion of the handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in accordance with a determination that the text entry user interface is operating in a first mode in which handwritten user input is converted to font-based text in response to selection of a selectable option displayed with the handwritten user input, the respective timing criteria have been met when one or more first time thresholds have elapsed since the end of the first sequence of one or more handwritten user inputs (1338), such as in
In some embodiments, in accordance with a determination that the text entry user interface is operating in a second mode in which handwritten user input is converted to font-based text without display or selection of a selectable option for doing so, the respective timing criteria have been met when one or more second time thresholds, less than the one or more first time thresholds, have elapsed since the end of the first sequence of one or more handwritten user inputs (1340), such as in
The above-described manner of converting handwritten inputs to text (e.g., by providing two modes of conversion, one in which inputs are converted as they are received and confirmed and one in which the entire handwritten input is converted after all inputs have been completed) allows the electronic device to convert according to two different conversion modes (e.g., by providing two conversion modes based on which mode is most appropriate for the situation), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing different conversion modes and deploying the mode that is more appropriate for the text insertion situation), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the first sequence of one or more handwritten user inputs corresponds to a first sequence of font-based text (1342), such as in
In some embodiments, in response to determining that the respective timing criteria have been met, the electronic device replaces (1346) the visual representation of the first sequence of one or more handwriting user inputs with the first sequence of font-based text, such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting handwritten text to the same resulting font-based text regardless of whether the conversion occurs as a result of satisfying a non-timer-based conversion criteria or as a result of the satisfaction of a timer-based conversion criteria) allows the electronic device to provide the user with consistent and reliable conversion of handwritten text (e.g., by ensuring that conversion without the use of a timer results in the same font-based text as timer-based conversion), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing different conversion modes and deploying the mode that is more appropriate for the text insertion situation), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the first sequence of one or more handwritten user inputs corresponds to a first sequence of font-based text (1348), such as in
In some embodiments, in response to determining that the respective timing criteria have been met, the electronic device replaces (1352) the visual representation of the first sequence of one or more handwriting user inputs are of font-based text, different than the first sequence of font-based text, such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting handwritten text to font-based text while simultaneously and automatically correcting identified errors in the handwritten text) allows the electronic device to automatically correct user errors in the handwritten text (e.g., by identifying errors in the handwritten text and automatically correct the errors during the process of converting the handwritten input to font-based text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically correcting errors in the user's handwritten input without requiring the user to perform additional inputs or navigate to a separate user interface to correct the errors after the conversion to font-based text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the first sequence of one or more handwritten user inputs corresponds to a first sequence of font-based text (1354), such as in
In some embodiments, in response to detecting the second sequence of one or more handwriting user inputs, the electronic device displays (1358), with the visual representation of the first sequence of one or more handwriting user inputs, a visual representation of the second sequence of one or more handwriting user inputs, such as in
In some embodiments, in response to determining that the respective timing criteria have been met (1362), the electronic device replaces (1364) the visual representation of the first sequence of one or more handwriting user inputs with the first sequence of font-based text, such as in
In some embodiments, in response to determining that the respective timing criteria have been met (1362), the electronic device replaces (1366) the visual representation of the second sequence of one or more handwriting user inputs with the second sequence of font-based text, such as in
The above-described manner of converting handwritten inputs to text (e.g., by converting a first sequence of handwritten input and a second sequence of handwritten input simultaneously based on a single timer) allows the electronic device to combine text conversion operations and reduce the disruption to the user (e.g., by converting the first and second sequence of handwritten inputs at the same time based on the timer for the first sequence of handwritten inputs or a timer that was reset when the second sequence of handwritten inputs was received), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by converting both sequences of handwritten input at the same time without requiring the user to wait for the conversion of both sequences of handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
Users interact with electronic devices in many different manners, including entering text into the electronic device. In some embodiments, an electronic device displays a user interface that accepts both textual and graphical inputs. The embodiments described below provide ways in which an electronic device displays input control menus for controlling user inputs into text fields that accept both textual and graphical inputs. Enhancing interactions with a device reduces the amount of time needed by a user to perform operations, and thus reduces the power usage of the device and increases battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally referred to as a user of the device.
In
In
As shown in
In some embodiments, selectable options 1414-1 to 1414-4 correspond to a plurality of drawing tools. In some embodiments, the drawing tools control the shape, size, style, and other visual characteristics of the handwritten input. For example, if selectable option 1414-1 corresponding to the text entry drawing tool is selected, then device 500 is in a text input mode such that handwriting inputs from stylus 203 are interpreted as requests to enter text and are thus converted into font-based text. In some embodiments, if selectable option 1414-2 corresponding to a pen drawing tool is selected, then device 500 is in a pen input mode such that handwriting inputs from stylus 203 are interpreted as a drawing and thus have the visual characteristics associated with drawing using a pen (e.g., medium sized lines). In some embodiments, if selectable option 1414-3 corresponding to a marker drawing tool is selected, then device is in a marker input mode such that handwriting inputs from stylus 203 are interpreted as a drawing and have the visual characteristics associated with drawing using a marker (e.g., thicker and optionally rectangular lines). In some embodiments, if selectable option 1414-4 corresponding to a pencil drawing tool is selected, then device is in a pencil input mode such that handwriting inputs from stylus 203 are interpreted as a drawing and have the visual characteristics associated with drawing using a pencil (e.g., thin lines). In some embodiments, more or fewer drawing tools can be displayed on handwriting entry menu 1410.
In some embodiments, selectable options 1416 are a set of options corresponding to the selected drawing tool (e.g., in
In
In
In
In
In
In some embodiments, handwriting entry menu 1420 includes selectable option 1422-1 corresponding to an undo option, which is selectable to undo the most recently performed function or operation. In some embodiments, handwriting entry menu 1420 includes selectable option 1422-2 corresponds to a redo option, which is selectable to redo the most recently undone function or operation, or to re-perform the most recently performed function or operation. In some embodiments, handwriting entry menu 1420 includes a set of color options 1424. In some embodiments, the set of color options 1424 include one or more selectable options for setting the color of the handwritten input. In some embodiments, a halo surrounding a particular color option indicates the color option that is currently selected (e.g., a halo around the block color option). In some embodiments, the set of color options 1424 includes a selectable option to display a color palette from which the user is able to select a desired color. In some embodiments, handwriting entry menu 1420 includes object insertion options 1426. For example, object insertion options 1426 includes a selectable option that is selectable to insert a text box into general entry region 1404. In some embodiments, object insertion options 1426 includes a selectable option that is selectable to insert a geometric shape (e.g., circles, square, triangles, lines, etc.) into general entry region 1404. In some embodiments, handwriting entry menu 1420 includes selectable option 1419 to re-display handwriting entry menu 1410. In some embodiments, handwriting entry menu 1420 can include more or fewer selectable options than those shown and discussed here.
In
In
In
As described below, the method 1500 provides ways to presenting handwritten entry menus. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, an electronic device (e.g., an electronic device, a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device) including a touch screen, or a computer including a touch screen, such as device 100, device 300, device 500, device 501, or device 591) in communication with a touch-sensitive display displays (1502), on the touch-sensitive display, a user interface including a first content entry region, such as in
In some embodiments, while displaying the user interface, the electronic device detects (1504), via the touch-sensitive display, a user input corresponding to a request to initiate content entry into the content entry region that includes detecting a contact in the content entry region, such as in
In some embodiments, in response to detecting the user input (1506), in accordance with a determination that the user input comprises input with a finger in a content entry region, the electronic device displays (1508), on the touch-sensitive display, a content entry user interface that includes a soft keyboard for entering text into the content entry region, such as in
In some embodiments, in response to detecting the user input (1506), in accordance with a determination that the user input comprises input with a stylus in the content entry region, the electronic device displays (1510), on the touch-sensitive display, the content entry user interface for generating content using the stylus without displaying a soft keyboard for entering (font-based) text into the content entry region, such as in
The above-described manner of providing content entry options (e.g., by displaying a content entry user interface that includes a soft keyboard when the input is received from a finger and displaying the content entry user interface without the soft keyboard when the input is received from a stylus) allows the electronic device to provide the user with a context specific menu for entering content into a content entry region (e.g., by determining that a virtual keyboard should be displayed if the user is using his or her finger to enter content, and by determining that no virtual keyboard should be displayed if the user is using a stylus (e.g., because handwritten input is optionally converted into computer text) and displaying the appropriate options accordingly), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with the appropriate options based on the user's input device without requiring the user to navigate to a separate menu or perform additional inputs to reach the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the content entry user interface that includes the soft keyboard for entering text into the content entry region, the electronic device detects (1512), via the touch-sensitive display, a second user input in the content entry region, such as in
The above-described manner of removing display of a soft keyboard (e.g., by receiving an input from a stylus and removing display of the soft keyboard) allows the electronic device to update the menu for entering content to remove the keyboard when it's no longer needed (e.g., by determining that a virtual keyboard is unnecessary if the user is using a stylus (e.g., because handwritten input is optionally converted into font-based text such that a soft keyboard is unnecessary)), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically providing the user with the appropriate options based on the user's switching to using a stylus without requiring the user to navigate to a separate menu or perform additional inputs to remove the soft keyboard), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the content entry user interface for generating content using the stylus without displaying the soft keyboard for entering text into the content entry region (e.g., while displaying the menu that is displayed when the user is interacting with the display with a stylus), the electronic device detects (1518), via the touch-sensitive display, a second user input in the content entry region, such as in
In some embodiments, in response to detecting the second user input (1520), in accordance with a determination that the second user input comprises input with a finger in the content entry region, the electronic device displays (1522), on the touch-sensitive display, the soft keyboard, such as in
The above-described manner of displaying a soft keyboard (e.g., by receiving an input from a finger and displaying the soft keyboard) allows the electronic device to update the menu for entering content to display the keyboard when it may be needed (e.g., by determining that a virtual keyboard is likely needed if the user is interacting with his or her finger (e.g., to enter text)), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically providing the user with a soft keyboard based on the user's switching to using his or her finger without requiring the user to navigate to a separate menu or perform additional inputs to display the soft keyboard), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in accordance with a determination that the content entry region satisfies one or more criteria, the content entry user interface for generating content using the stylus without displaying the soft keyboard for entering text into the content entry region includes one or more tools for controlling drawing content entry into the content entry region using the stylus (1524), such as in
The above-described manner of displaying a tools for controlling drawing from the stylus (e.g., by automatically displaying drawing options when the content entry region satisfies certain criteria (e.g., accepts drawing inputs)) allows the electronic device to update the menu based on the characteristic of the content entry region (e.g., by determining that the content entry region supports drawings and displaying options for the user to control drawing content), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically providing the user with the options that are available based on the compatibility of the content entry region without requiring the user to navigate to a separate menu or perform additional inputs to activate the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the content entry region satisfies the one or more criteria when the content entry region is capable of accepting drawing input, and does not satisfy the one or more criteria when the content entry region is not capable of accepting drawing input (1526), such as in
The above-described manner of displaying tools for controlling drawing from the stylus (e.g., by automatically displaying drawing options when the content entry region supports drawing options) allows the electronic device to update the menu based on the characteristic of the content entry region (e.g., by determining that the content entry region supports drawings and displaying options for the user to control drawing content), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically providing the user with the options that are available based on the compatibility of the content entry region without requiring the user to navigate to a separate menu or perform additional inputs to activate the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the content entry user interface for generating content using the stylus includes (1528): one or more tools for controlling drawing content entry into the content entry region using the stylus (1530) (e.g., a pencil tool, a pen tool, a highlighting tool, a marker tool, a charcoal tool, etc.); and a respective text entry tool for entering font-based text into the content entry region using handwritten input from the stylus (1532), such as in
The above-described manner of displaying tools for controlling input from the stylus (e.g., by displaying options for drawing and entering text when the content entry region supports entry of both drawings and text) allows the electronic device to update the menu based on the characteristic of the content entry region (e.g., by determining that the content entry region supports drawings and text and displaying options for the user to enter drawing content and text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically providing the user with the options that are available based on the compatibility of the content entry region without requiring the user to navigate to a separate menu or perform additional inputs to activate the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the content entry user interface for generating content using the stylus includes (1534): a first set of one or more tools, including the one or more tools, for controlling drawing content entry into the content entry region using the stylus (1536), such as in
The above-described manner of displaying sets of tools for controlling input from the stylus (e.g., by a selectable option to switch between a set first of tools and a second set of tools) allows the electronic device to provide multiple options and organize the options based on usage (e.g., by organizing tools into a first set or a second set of options and providing an option to switch between selecting from one set of options and selecting from a second set of options), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with multiple sets of the options that are available based on the compatibility of the content entry region and allowing the user to switch between the two sets without requiring the user to navigate to a separate menu or perform additional inputs to access the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the content entry user interface that includes the soft keyboard for entering text into the content entry region, the electronic device detects (1542), via the touch-sensitive display, an input corresponding to a request to cease display of the soft keyboard, wherein the soft keyboard is displayed with one or more selectable options for modifying text in the content entry region, such as in
In some embodiments, in response to receiving the input corresponding to the request to cease display of the soft keyboard (1544), the electronic device ceases (1546) display of the soft keyboard while maintaining display, in the user interface, of the one or more selectable options for modifying text in the content entry region, such as in
The above-described manner of maintaining display of options for modifying text (e.g., by displaying options for modifying text when a soft keyboard is shown and maintaining options for modifying text after the soft keyboard is dismissed) allows the electronic device to continue to provide the user with options for modifying text (e.g., by maintaining display of the options for modifying text even after the soft keyboard is dismissed when it is likely that the user will want the options (e.g., because the user is using a stylus to input text instead of the soft keyboard)), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by maintaining the options for modifying text when the user begins to enter text using a stylus without requiring the user to navigate to a separate menu or perform additional inputs to access the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the content entry user interface that includes the soft keyboard for entering text into the content entry region, wherein the soft keyboard includes one or more first keys and one or more second keys, the electronic device detects (1548), via the touch-sensitive display, an input corresponding to a request to cease display of the soft keyboard, such as in
In some embodiments, in response to receiving the input corresponding to the request to cease display of the soft keyboard (1550): the electronic device ceases (1552) display of the soft keyboard; and the electronic device displays (1554), in the user interface, one or more selectable options corresponding to the one or more first keys, such as in
The above-described manner of maintaining display of one or more selectable options (e.g., by relocating one or more options from the soft keyboard to the user interface of the application after the soft keyboard is dismissed) allows the electronic device to continue to provide the user with select keyboard options (e.g., by maintaining display of the options even after the soft keyboard is dismissed when it is likely that the user will want the options), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by maintaining the options when the user dismisses the keyboard but is still interacting with the user interface without requiring the user to navigate to a separate menu or perform additional inputs to access the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
As described below, the method 1600 provides ways to control the characteristics of handwritten input based on selections on a handwritten entry menu. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, an electronic device (e.g., an electronic device, a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device) including a touch screen, or a computer including a touch screen, such as device 100, device 300, device 500, device 501, or device 591) in communication with a touch-sensitive display displays (1602), on the touch-sensitive display, a content entry user interface, such as in
In some embodiments, while displaying the content entry user interface, the electronic device receives (1604), via the touch-sensitive display, a handwritten user input corresponding to the content entry user interface, such as in
In some embodiments, in response to receiving the handwritten user input (1606), in accordance with a determination that a text entry drawing tool was selected when the handwritten user input was detected, the electronic device initiates (1608) a process to convert the handwritten user input into a first sequence of font-based text characters, in the content entry user interface, corresponding to the handwritten user input, such as in
In some embodiments, in response to receiving the handwritten user input (1606), in accordance with a determination that a drawing tool other than the text entry drawing tool was selected when the handwritten input was detected, the electronic device displays (1610), in the content entry user interface, a visual representation of the handwritten user input without initiating the process to convert the handwritten user input into the first sequence of font-based text characters, such as in
The above-described manner of interpreting handwritten input (e.g., by converting handwritten user input to text if a text entry mode is active and not converting the handwritten user input if text entry mode is not active) allows the electronic device to provide the user with the ability to switch between writing text and not writing text (e.g., by converting handwritten input into text if the text entry mode is active or leaving the handwritten input unmodified if the text entry mode is not active), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to use the same handwritten input to enter text or draw an image by toggling the text entry mode without requiring the user to switch to a different input device or navigate to a separate user interface to switch between entering text and drawing an image), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in accordance with a determination that the text entry drawing tool is selected, the electronic device displays (1612), in the content entry user interface, one or more options for controlling formatting of font-based text in the content entry user interface, such as in
The above-described manner of presenting input options (e.g., by presenting font-based text formatting options when the text entry drawing tool is selected) allows the electronic device to provide the user with the most relevant options for the input operation that is selected (e.g., by presenting font-based text formatting options when the text entry drawing tool enables handwritten input to be converted into font-based text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically determining the options that are likely desired by the user without requiring the user to navigate to a separate user interface or perform additional inputs to access the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in accordance with a determination that a drawing tool other than the text entry drawing tool is selected, the electronic device displays (1614), in the content entry user interface, one or more options for controlling drawing input entry in the content entry user interface, such as in
The above-described manner of presenting input options (e.g., by presenting drawing input options when a drawing tool other than the text entry drawing tool is selected) allows the electronic device to provide the user with the most relevant options for the input operation that is selected (e.g., by presenting drawing options when a drawing tool is selected), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically determining the options that are likely desired by the user without requiring the user to navigate to a separate user interface or perform additional inputs to access the same options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the content entry user interface includes a selectable option to display a keyboard for entering font-based text in the content entry user interface (1616), such as in
The above-described manner of displaying a virtual keyboard (e.g., by presenting a selectable option to display a virtual keyboard) allows the electronic device to provide the user with the option to switch to entering text using a virtual keyboard (e.g., by presenting a selectable option to display a virtual keyboard to enter text), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to switch from using handwritten input to enter text to using a familiar virtual keyboard to enter text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in response to receiving the handwritten user input, the electronic device displays (1618), in the content entry user interface, the visual representation of the handwritten user input, such as in
In some embodiments, after displaying the visual representation of the handwritten user input in the content entry user interface (1620), in accordance with the determination that the text entry drawing tool was selected when the handwritten user input was detected, the electronic device ceases (1622) to display the visual representation of the handwritten user input in the content entry user interface, and converting the visual representation of the handwritten user input into font-based text, such as in
In some embodiments, after displaying the visual representation of the handwritten user input in the content entry user interface (1620), in accordance with the determination that the text entry drawing tool was not selected when the handwritten user input was detected, the electronic device maintains (1624) display of the visual representation of the handwritten user input in the content entry user interface without converting the visual representation of the handwritten user input into font-based text, such as in
The above-described manner of displaying handwritten input on the display (e.g., by always displaying the handwritten input as the input is received on the display regardless of the tool that is selected and only removing the handwritten input if it is converted into font-based text (e.g., when the text entry drawing tool is selected)) allows the electronic device to provide the user with visual feedback on the user's handwritten input (e.g., by displaying the handwritten input whenever the handwritten input is received, regardless of the tool that is selected, thus allowing the user to see what the user is inputting), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user feedback of the user's input whenever the user is performing handwritten input in the content entry user interface), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the visual representation of the handwritten user input displayed in accordance with the determination that a drawing tool other than the text entry drawing tool was selected when the handwritten input was detected comprises a line having a respective appearance (1626), such as in
In some embodiments, in accordance with a determination that the drawing tool is a first drawing tool, the respective appearance is a first appearance (1628), such as in
In some embodiments, in accordance with a determination that the drawing tool is a second drawing tool, different than the first drawing tool, the respective appearance is a second appearance, different than the first appearance (1630), such as in
The above-described manner of displaying handwritten input on the display (e.g., by displaying the handwritten input with different appearances based on the drawing tool that is selected) allows the electronic device to provide the user with options for mimicking different drawing utensils (e.g., by displaying the handwritten input with visual characteristics based on the particular drawing tool that was selected), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with the ability to mimic different drawing devices using the same input device without requiring the user to navigate to a separate user interface or use a separate input device to achieve different drawing styles), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
Users interact with electronic devices in many different manners, including entering text into the electronic device. The embodiments described below provide ways in which an electronic device accepts handwritten inputs from a handwriting input device (e.g., a stylus) and provides the user with autocomplete suggestions, thus enhancing the user's interactions with the device. Enhancing interactions with a device reduces the amount of time needed by a user to perform operations, and thus reduces the power usage of the device and increases battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally referred to as a user of the device.
In
In some embodiments, autocomplete suggestion 1706 is displayed with a different visual appearance than handwritten input 1704 (e.g., to indicate that autocomplete suggestion 1706 is a suggestion and has not been entered into text entry field). For example, in
As shown in
In
In some embodiments, in response to the continued handwritten input, autocomplete suggestion 1706 is updated to suggest new characters based on the new character(s) that the user has written, as shown in
As shown in
In
In
In
In some embodiments, accepting any portion of the autocomplete suggestion (e.g., by underlining or other gesture), causes the entire autocomplete suggestion to be accepted. In some embodiments, the user is able to accept a portion, but not other portions, of the autocomplete suggestion (e.g., a subset of the characters). For example, in
In
In
In
In
As described below, the method 1800 provides ways of presenting autocomplete suggestions. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, such as in
In some embodiments, while displaying the user interface, the electronic device receives (1804), via the touch-sensitive display, a first user input comprising a first handwritten input directed to the user interface (e.g., receiving a handwritten input on or near a text field), wherein the first handwritten input corresponds to a first sequence of characters, such as in
In some embodiments, the user input is received from a stylus or other writing device. In some embodiments, the user input is received from a finger. In some embodiments, the handwritten input is received at a location on or near the text field that is indicative of a request to enter text into the text entry field. For example, a handwritten input that begins in the text field optionally indicates that the entire sequence of handwritten inputs is intended to be entered into the text field, even if a portion of the handwritten input extends outside of the text field. In some embodiments, a user input that begins outside of the text field but a substantial amount of the handwritten input falls within the text field is optionally considered to be an intent to enter text into the text field (e.g., 30%, 50%, etc.). In some embodiments, the text entry field includes a predetermined margin of error in which handwritten inputs within a certain distance from the text entry field will be considered to be a handwritten input within the text entry field. In some embodiments, the first sequence of characters is a partially written word (e.g., an incomplete word).
In some embodiments, in response to receiving the first user input, the electronic device displays (1806), in the user interface, a representation of the first handwritten input (e.g., displaying a trail of the handwritten input on the display as the input is received) and a representation of one or more predicted characters selected based on the first sequence of characters in the first handwritten input (e.g., displaying concurrently with the handwritten input (e.g., aligned with the handwritten input), font-based text that corresponds to characters that if added to the first sequence of characters would complete a suggested word that is optionally displayed without displaying font-based text of the first portion of the suggested word), wherein the representation of the one or more predicted characters is displayed after the representation of the first handwritten input in a writing direction, such as in
In some embodiments, as the user “draws” on the touch-sensitive display, the display shows the user's handwritten input at the location where the input was received. In some embodiments, the handwritten input trail is shown wherever on the touch-sensitive display the handwritten input is received. In some embodiments, if the sequence of characters is a partially written word, then the electronic device displays suggested character(s) to complete the user's partially written word into a suggested word. In some embodiments, the one or more predicted characters are the remaining characters of a suggested word to the user (e.g., the characters that are to be added to the handwritten input to result in the predicted word). In some embodiments, the predicted characters are displayed after a pause in the handwritten input (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds). In some embodiments, the predicted word is determined based on one or more factors for predicting the user's desired word, such as popularity of usage by the user or a plurality of users (e.g., other than the user), the commonality of the word, the context of the sentence, etc.
The above-described manner of suggesting words to the user (e.g., by receiving a handwritten input and displaying the remainder of a suggested word to the user) allows the electronic device to provide the user with a suggested word (e.g., by displaying the remainder of the suggested word to the user), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by allowing the user to handwrite text and automatically determining the word that the user is most likely writing and suggesting the word to the user by displaying the remainder of the letters to the user), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, while displaying the representation of the first handwritten input and the representation of the one or more predicted characters, the electronic device receives (1808), via the one or more input devices, a second user input comprising a second handwritten input directed to the user interface, such as in
In some embodiments, in response to receiving the second user input comprising the second handwritten input directed to the user interface (1810), in accordance with a determination that the second handwritten input satisfies one or more first criteria with respect to the representation of the one or more predicted characters, the electronic device accepts (1812) the one or more predicted characters for use in (e.g., for entry into a text field displayed in) the user interface, such as in
In some embodiments, in response to receiving the second user input comprising the second handwritten input directed to the user interface (1810), in accordance with a determination that the second handwritten input does not satisfy the one or more first criteria with respect to the representation of the one or more predicted characters, the electronic device forgoes (1814) accepting the one or more predicted characters for use in (e.g., for entry into a text field displayed in) the user interface, such as in
The above-described manner of accepting predicted characters (e.g., by receiving a handwritten input directed at the predicted characters and accepting the predicted characters if the handwritten input satisfies a first criteria) enables the suggested word to be accepted with a quick gesture, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically suggesting words to the user in line with the user's writing and providing the user with an easy method of accepting the suggested word without navigating to a separate user interface or perform additional inputs to accept the suggested word), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, while displaying the representation of the first handwritten input and the representation of the one or more predicted characters, the electronic device receives (1816), via the one or more input devices, a second user input comprising a second handwritten input directed to the user interface, such as in
In some embodiments, in response to receiving the second user input comprising the second handwritten input directed to the user interface (1818), in accordance with a determination that the second handwritten input satisfies one or more first criteria with respect to a first portion of the representation of the one or more predicted characters but not a second portion of the representation of the one or more predicted characters, the electronic device accepts (1820) a subset of the one or more predicted characters corresponding to the first portion of the representation of the one or more predicted characters for use in (e.g., for entry into a text field displayed in) the user interface, such as in
In some embodiments, the one or more characters that have been underlined are updated to have a different visual characteristic (e.g., change color, change opacity, etc.) to indicate to the user which characters the user has accepted. For example, the one or more predicted character are displayed as grey text and as the user underlines the characters, the underlined characters become black indicating that the user has accepted that character. In some embodiments, the second portion of the one or more predicted characters cease to be displayed after the first portion has been entered into the text entry region as inputs. In some embodiments, when the first portion of the one or more predicted characters are entered into the text entry region, the handwritten input is converted to font-based text and the first portion of the characters is aligned with the font-based text corresponding to the handwritten input.
In some embodiments, in response to receiving the second user input comprising the second handwritten input directed to the user interface (1818), in accordance with a determination that the second handwritten input satisfies one or more first criteria with respect to the first and second portions of the representation of the one or more predicted characters, the electronic device accepts (1822) a portion of the one or more predicted characters corresponding to the first and second portions of the representation of the one or more predicted characters for use in (e.g., for entry into a text field displayed in) the user interface, such as in
The above-described manner of accepting predicted characters (e.g., by receiving a handwritten input underlining the characters that the user wants to accept) enables a portion of the suggested word to be accepted with a quick gesture, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with an easy method of accepting the suggested word without navigating to a separate user interface or perform additional inputs to accept the suggested word), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, accepting one or more respective predicted characters for use in (e.g., for entry into a text field displayed in) the user interface includes (1824), ceasing to display the representation of the first handwritten input and a representation of the one or more respective predicted characters (1826), such as in
In some embodiments, accepting one or more respective predicted characters for use in (e.g., for entry into a text field displayed in) the user interface includes (1824), displaying, in the user interface, a representation of (1828), the first sequence of characters corresponding to the first handwritten input (1830), and the one or more respective predicted characters (1832), such as in
The above-described manner of accepting predicted characters (e.g., by replacing both the handwritten input and the predicted characters with font-based text of the combination of the handwritten input and the accepted predicted characters) enables the suggested word to be used in the user interface with a quick gesture, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by converting the handwritten input into font-based text at the same time that the predicted characters are entered into the user interface without requiring the user to wait for the handwritten input to be converted into font-based text separately from accepting the predicted characters), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, while receiving the second user input that satisfies the one or more first criteria with respect to the representation of the one or more predicted characters, the electronic device changes (1834) a value of a display characteristic of respective ones of the one or more predicted characters as the second user input satisfies the one or more first criteria for the respective ones of the one or more predicted characters, such as in
The above-described manner of accepting predicted characters (e.g., by changing the visual characteristic of the characters that have so-far been selected) allows the electronic device to provide confirmation about what characters have been accepted and will be entered, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., providing a live visual indicator of which characters the user has selected), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device, while reducing errors in the usage of the device.
In some embodiments, while displaying the representation of the first handwritten input and the representation of the one or more predicted characters, the electronic device receives (1836), via the one or more input devices, a second user input comprising a second handwritten input directed to the user interface, such as in
In some embodiments, in response to receiving the second user input comprising the second handwritten input directed to the user interface (1838), in accordance with a determination that the second handwritten input comprises a continuation of the first handwritten input, the electronic device ceases (1840) display of the representation of at least a subset of the one or more predicted characters, such as in
The above-described manner of rejecting suggested characters (e.g., by ceasing display of the predicted characters when the user continues handwritten input indicating that the user does not want to accept the predicted characters) enables continued handwritten input to be provided without interruption, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically ceasing display of the characters when the user continues handwritten input without requiring the user to perform additional inputs to dismiss the display of the predicted characters), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in response to receiving the second user input comprising the second handwritten input directed to the user interface (1842) (e.g., further handwritten inputs), in accordance with the determination that the second handwritten input comprises the continuation of the first handwritten input, the electronic device displays (1844), in the user interface, a representation of the first handwritten input and the second handwritten input, wherein the second handwritten input corresponds to a second sequence of characters (1846), such as in
In some embodiments, a combination of the first sequence of characters, the second sequence of characters, and the one or more second predicted characters is different than a combination of the first sequence of characters and the one or more predicted characters (1850), such as in
The above-described manner of updating the displayed predicted characters (e.g., by changing the displayed predicted characters based on further handwritten inputs) allows the electronic device to provide updated predicted words based on further handwritten input, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically updating the suggested words in accordance with the user's input to continually provide the user with relevant predicted words without requiring the user to perform an additional input to update the predicted words), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in response to receiving the second user input comprising the second handwritten input directed to the user interface (1852) (e.g., further handwritten input), in accordance with the determination that the second handwritten input comprises the continuation of the first handwritten input, the electronic device displays (1854), in the user interface, a representation of the first handwritten input and the second handwritten input, wherein the second handwritten input corresponds to a second sequence of characters (1856), such as in
In some embodiments, a combination of the first sequence of characters, the second sequence of characters, and the one or more second predicted characters is the same as a combination of the first sequence of characters and the one or more predicted characters (1860), such as in
The above-described manner of updating predicted characters (e.g., by updating the predicted characters to remove display of the characters that the user's further handwritten input has written) allows the electronic device to provide the continued ability to accept the suggested word even as the user continues to write the suggested word, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically removing display of the characters that the user has written as the user writes it, without requiring the user to see irrelevant characters that the user is no longer interested in anymore (e.g., because the user has already written them)), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the representation of the first handwritten input and the representation of the one or more predicted characters (1862), in accordance with a determination that one or more criteria are satisfied, the electronic device displays (1864), in the user interface, an animation of a representation of a handwritten input for accepting the one or more predicted characters for use in (e.g., for entry into a text field displayed in) the user interface, such as in
In some embodiments, while displaying the representation of the first handwritten input and the representation of the one or more predicted characters (1862), in accordance with a determination that the one or more criteria are not satisfied, the electronic device forgoes (1866) displaying, in the user interface, the animation of the representation of the handwritten input for accepting the one or more predicted characters for use in (e.g., for entry into a text field displayed in) the user interface, such as in
The above-described manner of displaying a hint of how to accept predicted characters (e.g., by displaying an underlining animation underlining the predicted characters) provides a visual indication of a gesture for accepting suggested words, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically displaying a short tutorial of how to accept predicted words without requiring the user to perform separate research to determine how to accept predicted words), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more criteria include a criterion that is satisfied when the electronic device has detected the handwritten input for accepting predicted characters for use in the user interface fewer than a threshold number of times (e.g., has never detected the handwritten input for accepting predicted characters, has detected the handwritten input for accepting predicted characters less than five times or another predetermined number of times), and is not satisfied when the electronic device has not detected the handwritten input for accepting predicted characters for use in the user interface at least the threshold number of times (1868), such as in
The above-described manner of limiting display of the hint of how to accept predicted characters (e.g., by no longer displaying the animation if the user has previously performed the gesture to accept predicted characters indicating that the user knows how to accept predicted characters) allows the electronic device to avoid unnecessarily displaying animations on the display, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically determining that the user likely does not need a hint to learn how to accept predicted characters and forgoing displaying the hint in the future), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the first handwritten input is directed to a first text entry region in the user interface, the one or more criteria include a criterion that is satisfied when the electronic device has displayed predicted characters in the first text entry region fewer than a threshold number of times (e.g., has never displayed predicted characters in the first text entry region, has displayed predicted characters in the first text entry region fewer than five times or another predetermined number of times), and is not satisfied when the electronic device has not displayed predicted characters in the first text entry region at least the threshold number of times (1870), such as in
The above-described manner of limiting display of the hint of how to accept predicted characters (e.g., by only displaying the animation one time for each text entry region) allows the electronic device to indicate that the current text entry region supports accepting predicted characters while avoid unnecessarily displaying animations on the display, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying the hint once for each text entry region and forgoing displaying the hint for that text entry region in the future), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more criteria include a criterion that is satisfied when the electronic device has displayed predicted characters in the user interface fewer than a threshold number of times (e.g., has never displayed predicted characters in the user interface, has displayed predicted characters in the user interface fewer than five times or another predetermined number of times), and is not satisfied when the electronic device has not displayed predicted characters in the user interface at least the threshold number of times (1872), such as in
The above-described manner of limiting display of the hint of how to accept predicted characters (e.g., by only displaying the animation one time for each user interface) allows the electronic device to indicate that the current user interface supports accepting predicted characters while avoid unnecessarily displaying animations on the display, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying the hint once for each user interface and forgoing displaying the hint for that user interface in the future), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the one or more criteria include a criterion that is satisfied when the electronic device has displayed predicted characters during a current day fewer than a threshold number of times (e.g., has never displayed the predicted characters during the current day, has displayed predicted characters less than five times or another predetermined number of times during the current day), and is not satisfied when the electronic device has not displayed predicted characters during the current day at least the threshold number of times (1874), such as in
The above-described manner of limiting display of the hint of how to accept predicted characters (e.g., by only displaying the animation one time per day) allows the electronic device to provide a reminder of how to accept predicted characters while avoid unnecessarily displaying animations on the display, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying the hint once per day and forgoing displaying the hint for the rest of the day), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in accordance with a determination that a size of handwritten characters that make up the first handwritten input is a first size, the one or more predicted characters are displayed at a second size (1876), such as in
In some embodiments, in accordance with a determination that a size of the handwritten characters that make up the first handwritten input is a third size, different than the first size, the one or more predicted characters are displayed at a fourth size, different than the second size (1878), such as in
The above-described manner of displaying predicted characters (e.g., by displaying the predicted character with a respective size that is based on the size of the handwritten input) allows the electronic device to adjust the size of the predicted characters based on the size of the handwritten input to increase the continuity of the characters displayed on the display, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying the predicted characters with a respective size that is based on the size of the handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the second size matches the first size, and the fourth size matches the third size (1880), such as in
The above-described manner of displaying predicted characters (e.g., by matching the size of the predicted character with the size of the handwritten input) allows the electronic device to adjust the size of the predicted characters based on the size of the handwritten input to increase the continuity of the characters displayed on the display, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying the predicted characters with a respective size that matches the size of the handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the user interface, the electronic device receives (1882), via the touch-sensitive display, a second user input comprising a second handwritten input directed to the user interface, wherein the second handwritten input corresponds to a second sequence of characters, such as in
In some embodiments, in response to receiving the second user input (1884), in accordance with a determination that the second sequence of characters satisfies one or more criteria, the electronic device displays (1886), in the user interface, a representation of the second handwritten input and a representation of one or more second predicted characters selected based on the second sequence of characters in the second handwritten input, such as in
In some embodiments, in response to receiving the second user input (1884), in accordance with a determination that the second sequence of characters does not satisfy the one or more criteria, the electronic device displays (1888), in the user interface, the representation of the second handwritten input without displaying the representation of the one or more second predicted characters, such as in
The above-described manner of displaying predicted characters (e.g., by displaying predicted characters if the handwritten input satisfies a certain criteria (e.g., the handwritten input corresponds to a unique word)) allows the electronic device to limit the instances in which predictions are provided to the user and avoid providing predictions when the chances that the user will accept the prediction are low, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying predicted characters in certain circumstances when a user is more likely to accept the predicted characters), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the user interface, the electronic device receives (1890), via the touch-sensitive display, a second user input comprising a second handwritten input directed to the user interface, wherein the second handwritten input corresponds to a second sequence of characters, such as in
In some embodiments, in response to receiving the second user input (1892), the electronic device displays (1894), in the user interface, a representation of the second handwritten input, such as in
In some embodiments, in response to receiving the second user input (1892), in accordance with a determination that more than a predetermined amount of time has elapsed since an end of the second handwritten input, the electronic device displays (1896), in the user interface, a representation of one or more second predicted characters selected based on the second sequence of characters in the second handwritten input, such as in
In some embodiments, in response to receiving the second user input (1892), in accordance with a determination that less than the predetermined amount of time has elapsed since the end of the second handwritten input, the electronic device forgoes displaying (1898) the representation of the one or more second predicted characters, such as in
The above-described manner of providing predicted characters (e.g., by displaying the predicted character after the user has paused handwriting input for a threshold amount of time) allows the electronic device to provide predicted characters in a situation in which the user is more likely to see and consider the predicted characters while avoiding displaying the predicted characters while the user is actively performing handwritten input, which could unnecessarily distract the user, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient, which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, a combination of the first sequence of characters and the one or more predicted characters is displayed, in the user interface, in a selectable user interface element that is selectable to enter the combination of the first sequence of characters and the one or more predicted characters for use in (e.g., for entry into a text field displayed in) the user interface (1898-02), such as in
The above-described manner of displaying predicted characters (e.g., by displaying the predicted character in a pop-up near the location of the handwritten input) allows the electronic device to provide predicted words without blocking the user interface where the handwritten input is being detected, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by displaying the predicted characters in a pop-up where the user can see the predicted input while simultaneously providing handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the selectable user interface element includes a first representation of the combination of the first sequence of characters and the one or more predicted characters, and a second representation of the first sequence of characters, wherein the first representation is selectable to enter the combination of the first sequence of characters and the one or more predicted characters for use in (e.g., for entry into a text field displayed in) the user interface, and the second representation is selectable to enter the one or more predicted characters without the first sequence of characters for use in (e.g., for entry into a text field displayed in) the user interface (1898-04), such as in
The above-described manner of displaying predicted characters concurrently with font-based text corresponding to the handwritten input (e.g., by concurrently displaying the predicted character and the font-based text interpretation of the handwritten input in a pop-up) allows the electronic device to provide the ability to confirm the user's writing and accept a predicted word or accept the handwriting input as written so-far, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user with the option to accept the predicted word or accept the font-based text of what the user has written so far, without requiring the user to navigate to different user interfaces to select the predicted word or accept the handwriting input as written), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
Users interact with electronic devices in many different manners, including entering text into the electronic device. The embodiments described below provide ways in which an electronic device converts handwritten inputs into font-based text, thus enhancing the user's interactions with the device. Enhancing interactions with a device reduces the amount of time needed by a user to perform operations, and thus reduces the power usage of the device and increases battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally referred to as a user of the device.
In
In
In
In
In
In
In
In some embodiments, concurrently with or after handwritten input 1904 is fully converted to grey (e.g., and in response to the determination that the user has completed writing the word “handwriting”), device 500 begins the process of converting handwritten input 1904 into font-based text. In some embodiments, the process of converting handwritten input 1904 includes an animation transforming handwritten input 1904 into font-based text. In some embodiments, the animation includes dissolving a portion of handwritten input 1908, as shown in
In
In
In
For example, in
In
In
In
In some embodiments, concurrently with or after handwritten input 1904 is fully converted to grey (e.g., and in response to the determination that the user has completed writing the word “handwriting”), device 500 begins the process of converting handwritten input 1904 into font-based text. In some embodiments, the process of converting handwritten input 1904 includes an animation transforming handwritten input 1904 into font-based text. In some embodiments, the animation includes shrinking handwritten input 1908 to and/or towards the final size of the resulting font-based text and/or fading handwritten input 1908 out of view while concurrently fading the resulting font-based text into view. For example, in
It is understood that although
It is also understood that the embodiments described herein with respect to the animation of the handwritten input changing visual characteristics as the user writes is optionally performed any or every time handwritten input writing characters and/or words is received (e.g., as described above with respect to any of
As described below, the method 2000 provides ways to convert handwritten input to font-based text. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, such as in
In some embodiments, while continuing to detect the input (2004) (e.g., while the contact is maintained on the display), in response to detecting the input, the electronic device displays (2006), via the display device, a representation of the path with a first appearance at a first location in the user interface, such as in
In some embodiments, while continuing to detect the input (2004), after displaying the representation of the path with the first appearance, the electronic device changes (2008) an appearance of at least a portion of the representation of the path to a second appearance that is different from the first appearance, such as in
In some embodiments, after changing the appearance of the portion of the representation of the path to a second appearance that is different from the first appearance (2010) (e.g., and in response to detection of an animation criteria such as lift off of a contact corresponding to the input or detection of a word or character corresponding to the path), the electronic device displays (2012) one or more font-based characters that are selected based on the path at a second location in the user interface, such as in
In some embodiments, after changing the appearance of the portion of the representation of the path to a second appearance that is different from the first appearance (2010), the electronic device displays (2014) an animation of the portion of the path moving from the first location in the user interface to the second location in the user interface, such as in
The above-described manner of changing the appearance of the representation of the handwritten input and then displaying the animation of the path moving from its current location to the location of the font-based characters indicates which parts of the handwritten input will convert into font-based text and indicate what the font-based text will be, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing feedback about the operation that is about to occur), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the first appearance is a first color (e.g., black) and the second appearance is a second color (e.g., grey) that is different from the first color (2016), such as in
In some embodiments, changing the appearance of the portion of the representation of the path includes gradually animating a change in the appearance of the portion of the representation of the path by progressively changing sub-portions of the representation of the path from the first appearance to the second appearance in a direction determined based on the direction in which the representation of the path was initially displayed (2018), such as in
The above-described manner of changing the appearance of the representation of the handwritten input indicates to which part of the previously input handwritten input additional handwritten input can be added, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing feedback about the proper location of additional handwritten input), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, changing the appearance of the portion of the representation of the path includes gradually animating a change in the appearance of the portion of the representation of the path by progressively changing sub-portions of the representation of the path from the first appearance to the second appearance at a rate determined based on a speed at which the representation of the path was initially displayed (2020), such as in
The above-described manner of changing the appearance of the representation of the handwritten input based on the speed of the handwritten input ensures that the presentation of the feedback is not a bottleneck to receiving further input, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by reducing the time needed to provide the path feedback described above), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, changing the appearance of the portion of the representation of the path includes ceasing to animate the change in the appearance of the portion of the representation of the path from the first appearance to the second appearance when the portion of the representation of the path reaches a first threshold distance from the input that caused the path to be generated (2022), such as in
The above-described manner of not changing the appearance of the representation of the handwritten input in the portion of the representation closest to the current stylus location indicates that further handwritten input can still be accepted and incorporated with the previously detected handwritten input, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient, which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, after ceasing to animate the change in the appearance of the portion of the representation of the path, the electronic device detects (2024) continued movement of the input, such as in
The above-described manner of resuming the animation of the change in appearance provides for continued feedback with respect to additional handwritten input, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by not requiring any action other than continued handwritten input to continue providing feedback), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the animation of the portion of the path moving from the first location in the user interface to the second location in the user interface is displayed in response to detecting an end of the input (2028), such as in
The above-described manner of not animating the path to the second location until liftoff prevents the device from needlessly presenting the animation and erroneously ceasing display of the path, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by reducing the likelihood of ceasing display of the path too soon while additional handwritten input directed to the path may be detected), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the animation of the portion of the path moving from the first location in the user interface to the second location in the user interface is displayed in response to detecting that character recognition criteria have been met (2030), such as in
The above-described manner of not animating the path to the second location until character recognition criteria have been met prevents the device from needlessly presenting the animation and erroneously ceasing display of the path, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by reducing the likelihood of ceasing display of the path too soon while additional handwritten input directed to the path may be detected), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the animation of the portion of the path moving from the first location in the user interface to the second location in the user interface includes replacing the portion of the path with a plurality of separate particles that move relative to each other (e.g., toward each other or away from each other) as they move toward the second location (2032), such as in
The above-described manner of animating the path moving from the first location to the second location provides immediate feedback about which part of the handwriting corresponds to the font-based characters, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by visually associating the handwriting input with the final corresponding font-based text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the number of separate particles in the plurality of separate particles is determined at least in part based on a length of the portion of the representation of the path to which the plurality of separate particles correspond (2034), such as in
The above-described manner of utilizing more or fewer particles based on the length of the portion of the handwritten path provides immediate feedback about which part of the handwritten path corresponds to which character, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by visually associating portions of the handwritten input with portions of the font-based characters), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the animation of the portion of the path moving from the first location in the user interface to the second location in the user interface includes ceasing to display the animation before visual elements corresponding to the animation reach the second location (2036), such as in
In some embodiments, the one or more font-based characters include a sequence of font-based characters (2038), such as in
The above-described manner of performing character-by-character animation provides immediate feedback about which character in the handwritten path corresponds to which character in the font-based characters, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by visually associating characters in the handwritten input with characters in the font-based characters, which makes potential errors in the conversion clear), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
In some embodiments, the one or more font-based characters include a sequence of font-based words (2042), such as in
The above-described manner of performing word-by-word animation provides immediate feedback about which word in the handwritten path corresponds to which word in the font-based characters, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by visually associating words in the handwritten input with words in the font-based characters, which makes potential errors in the conversion clear), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency while reducing errors in the usage of the device.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
Users interact with electronic devices in many different manners, including entering text and drawings into the electronic device. In some embodiments, an electronic device provides a content entry palette which includes options for controlling content inserted into content entry regions. The embodiments described below provide ways in which an electronic device dynamically displays different tools and options in the content entry palette based on the current context of the content entry. In some embodiments, displaying different tools and options customizes the user's experience, thus enhancing interactions with the device. Enhancing interactions with a device reduces the amount of time needed by a user to perform operations, and thus reduces the power usage of the device and increases battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally referred to as a user of the device.
In some embodiments, user interface 2100 corresponds to a note taking application (e.g., similar to user interface 800 described above with respect to
In
As shown in
In some embodiments, content entry palette 2110 includes text entry tool 2114-1, pen entry tool 2114-2 and marker tool 2114-3. In some embodiments, more or fewer content entry tools can be included in content entry palette 2110. In some embodiments, selection of text entry tool 2114-1 causes the device to enter into text entry mode in which handwritten inputs drawn in the content entry region are analyzed for text characters, identified, and converted into font-based text (such as described above with respect to methods 700, 1100, 1300, 1500, 1600, 1800, and/or 2000). In some embodiments, selection of pen entry tool 2114-2 causes the device to enter into a pen entry mode in which handwritten inputs drawn in the content entry region are stylized as if drawn by a pen (e.g., without converting them to font-based text). In some embodiments, selection of marker entry tool 2114-3 causes the device to enter into a marker entry mode in which handwritten inputs drawn in the content entry region are stylized as if drawn by a marker (e.g., without converting them to font-based text). In some embodiments, content entry tools other than the text entry tool are referred to as drawing tools (e.g., because the tools allow a user to draw in the content entry region and are not converted into font-based text).
In some embodiments, one or more of the options included in content entry palette 2110 depend on the currently active content entry tool. In some embodiments, one or more options included in the content entry palette 2110 are displayed due to being associated with the currently active content entry tool. For example, in
In
In
In
In
Thus, in some embodiments, as described above, one or more of the options displayed in content entry palette 2110 are based on the content entry tool that is selected. In some embodiments, if a text entry tool is currently active, then content entry palette 2110 includes tools related to the entry of font-based text. In some embodiments, if a drawing tool is currently active (e.g., pen tool, marker tool, highlighter tool, etc.), then content entry palette 2110 includes tools related to the entry of drawings.
In
In
In
In
In
In
In
In some embodiments, additionally or alternatively to displaying options based on application, content entry palette 2110 is able to be displayed in a smaller mode based on the width of user interface 2101 (e.g., as a result of being in multitasking mode in which multiple applications are concurrently displayed). In some embodiments, when content entry palette 2110 is in a smaller mode, fewer options are displayed in content entry palette 2110. In some embodiments, when content entry palette 2110 is in a smaller mode, certain options are collapsed with other options and displayed in a pop-up).
In
Thus, as described above, device 500 is able to display different sets of options in content entry palette 2110 based on the application for which the palette is displayed, the type of content entry region for which content is being entered, and/or the size of the palette (which is optionally based on the width of the user interface).
In
As described below, the method 2200 provides ways to display options in a content entry palette. The method reduces the cognitive burden on a user when interacting with a user interface of the device of the disclosure, thereby creating a more efficient human-machine interface. For battery-operated electronic devices, increasing the efficiency of the user's interaction with the user interface conserves power and increases the time between battery charges.
In some embodiments, an electronic device in communication with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, a smartphone, a media player, or a wearable device), or a computer, optionally in communication with one or more of a mouse (e.g., external), trackpad (optionally integrated or external), touchpad (optionally integrated or external), remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), and/or a controller (e.g., external), etc.) displays (2202), via a display generation component, a user interface including a first content entry region (e.g., a region in the user interface in which a user is able to input and/or enter text, images, multimedia, etc.) and a content entry user interface element, such as content entry region 2102 and content entry palette 2110 in
For example, in an email composition user interface, a content entry region for the body of the email is capable of receiving (and transmitting over email) text, still images, videos, attachments, etc., such as described above with respect to user interface 1400. In another example, in a note taking application, a content entry region is capable of receiving handwritten text, drawings, figures, etc. and capable of inserting images, drawings, etc., such as described above with respect to user interface 620, 800, 1000, and 1210.
In some embodiments, the palette includes one or more representations of handwriting devices that correspond to different content entry modes (which are selectable to enter the respective content entry mode). In some embodiments, the palette includes options for changing the color, size, shape, font, etc. of the inserted handwritten content. In some embodiments, the palette includes options for inserting files, attachments, images, font-based text, etc., such as discussed above with respect to method 1500.
For example, while the electronic device is in a handwriting text entry mode, the device is able to receive handwritten inputs, recognize the handwritten inputs, and convert the handwritten input into font-based text (e.g., in a manner similar to the processes described above with respect to method 700, 1100, 1300, 1500, 1600, 1800, and/or 2000). In some embodiments, the handwritten input is received from a stylus, finger, or any other writing device. In some embodiments, only handwritten inputs from a stylus are converted into font-based text. In some embodiments, the first set of options corresponding to the first content entry mode includes one or more of a table entry option (for inserting a table in the content entry region), a font option (for changing the font of the font-based text), a checkbox entry option (for inserting an option button in the content entry region), a virtual keyboard option (for displaying a soft or virtual keyboard in the user interface), a camera option (for taking an image using a camera of the electronic device and inserting the image into the content entry region), a file attachment option (for inserting and/or attaching a file to the content entry region), an emoji option (for inserting an emoji into the content entry region), a copy/paste option (for copying content to a clipboard or inserting content from a clipboard), etc.
In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touch screen display), external display such as a monitor, projector, television, or a hardware component (optionally integrated or external) for projecting a user interface or causing a user interface to be visible to one or more users, etc. In some embodiments, the display generation component is a hardware component (e.g., including electrical components) capable of receiving display data and displaying a user interface.
In some embodiments, while displaying the content entry user interface element while the electronic device is in the first content entry mode, the electronic device receives (2204) a user input corresponding to a request to switch the electronic device from the first content entry mode to a second content entry mode in which the electronic device is configured for receiving handwritten input without converting the handwritten input into font-based text, such as selection pen entry tool 2114-2 in
For example, the user input selects a representation of a pencil, pen, marker, highlighter, etc. In some embodiments, the user input corresponds to a request to exit the handwriting text entry mode and enter another content entry mode associated with the selected representation. In some embodiments, the content entry modes other than the handwriting text entry mode do not convert handwritten inputs into font-based text. In some embodiments, handwritten input while in the content entry modes other than the handwriting text entry mode causes inserting of content based on the selected handwriting device. For example, while in pencil content entry mode (e.g., when the pencil tool is selected), the handwritten input is displayed with a style corresponding to a pencil drawing. In some embodiments, the handwritten input is received from a stylus, finger, or any other writing device. In some embodiments, the user input corresponding to a request to switch the electronic device is received via a stylus or touch input (e.g., selecting a respective tool on the content entry user interface element), a voice command (e.g., via a microphone), or any other suitable input mechanism.
In some embodiments, in response to receiving the user input (2206), such as in
In some embodiments, one or more options are removed from the content entry user interface element. In some embodiments, one or more options are added to the content entry user interface element. In some embodiments, the options that are removed do not apply to or are irrelevant to the second content entry mode. In some embodiments, the options that are added do not apply to or are irrelevant to the first content entry mode, but apply to and/or are relevant to the second content entry mode. For example, while in the handwriting text entry mode, the content entry user interface element includes a font option (e.g., selectable to change the font of the resulting font-based text, such as font size, font type, color, underline, italics, strike-through, subscript, superscript, etc.), and entering the pencil drawing content entry mode causes the font option to be removed from display and one or more color input options to be displayed (e.g., selectable to change the color of the inserted content and/or handwritten drawing). In some embodiments, in response to receiving the user input, the device is configured to operate in the second content entry mode. For example, if a user selects a drawing tool from the content entry user interface element, then the device enters into drawing mode and handwritten inputs are interpreted as a drawing and the inputs are not converted into computer text (e.g., font-based text).
The above-described manner of providing different content entry options for two content entry modes that are both based on handwritten input (e.g., using a stylus) but operate differently allows the electronic device to provide the user with options tailored for the content entry mode that the user is in, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically providing the user with options that are relevant to the active content entry mode and not providing the user with options that are irrelevant to the active content entry mode, without requiring the user to navigate to a separate menu or perform additional inputs to access the relevant options), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while the electronic device is in the first content entry mode, the electronic device receives (2210), via the one or more input devices, a user input comprising a handwritten input directed to the first content entry region, such as touch down of stylus 203 in
In some embodiments, in response to receiving the user input, the electronic device displays (2212) a representation of the handwritten input in the user interface at a location corresponding to the first content entry region, such as representation 2106 of the handwritten input in
In some embodiments, after displaying the representation of the handwritten input at the location corresponding to the first content entry region (2214), such as in
The above-described manner of converting handwritten inputs to text (e.g., by receiving the input directed to the first content entry region and replacing the handwritten input with font-based text if when the device is in the first content entry mode) allows the user to write directly onto the user interface to enter text if the text entry tool is selected, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., without requiring the user to switch to a different input mechanism such as a physical or virtual keyboard to switch between text entry mode and drawing mode), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while the electronic device is in the second content entry mode, the electronic device receives (2200), via the one or more input devices, a user input comprising a handwritten input directed to the first content entry region, such as touch down of stylus 203 in
In some embodiments, in response to receiving the user input, the electronic device displays (2222) a representation of the handwritten input in the user interface at a location corresponding to the first content entry region, without displaying font-based text corresponding to the representation of the handwritten input, such as representation of the handwritten input 2108 in
For example, as the user “draws” on the touch-sensitive display, the display shows the user's handwritten input at the location where the input was received (e.g., in the first content entry region). In some embodiments, the representation of the handwritten input is not replaced with font-based text while in the second content entry mode. In some embodiments, the second content entry mode is a drawing mode. In some embodiments, the second content entry mode is a content entry mode other than the text entry mode (e.g., a tool other than the text entry tool is selected), such as described above with respect to methods 1500 and 1600.
The above-described manner of accepting handwritten input as a drawing (e.g., by receiving the input directed to the first content entry region and not replacing the input with font-based text if the device is in the second content entry mode) allows the user to quickly and efficiently switch to a drawing mode and draw in the user interface using the same input device that is used to input font-based text (e.g., without requiring the user to switch to another input device or input mechanism), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient, which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the first set of options corresponding to the first content entry mode (2224), such as options 2116 in
In some embodiments, the options affect the visual characteristics of the representations of the handwritten input (e.g., future inputs) while the device is in the first content entry mode. In some embodiments, if text in the first content region is selected (e.g., highlighted), then the options affect the visual characteristics of the selected text. In some embodiments, the font settings includes font size, font type, bold, italics, underline, strikethrough states, color, etc. In some embodiments, a soft or virtual keyboard is a visual representation of a physical keyboard. In some embodiments, user selection of characters on the soft keyboard causes the respective characters to be entered into the first content entry region.
The above-described manner of displaying options associated with the text entry mode in the content entry user interface element (e.g., by including options specific to font-based text when the device is in text entry mode) allows the user to quickly and efficiently configure the font-based text that is entered in the content entry region, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., without requiring the user to navigate to a separate user interface or perform additional inputs to change the visual characteristics of the font-based text), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the second set of options corresponding to the second content entry mode includes one or more options for selecting one or more color settings for representations of handwritten input in the first content entry region (2232), such as options 2113 in
The above-described manner of displaying options associated with drawing mode in the content entry user interface element (e.g., by including options specific to drawings when the device is in drawing mode) allows the user to quickly and efficiently configure the drawings that are entered in the content entry region, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., without requiring the user to navigate to a separate user interface or perform additional inputs to change the visual characteristics of the drawings), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, while displaying the content entry user interface element, the electronic device receives (2234) a user input directed to the content entry user interface element, such as in
In some embodiments, in response to receiving the user input (2236), in accordance with a determination that the user input includes a selection input directed to a location corresponding to the content entry user interface element and a movement while maintaining the selection input, the electronic device moves (2238) the content entry user interface element within the user interface in accordance with the movement of the user input, such as in
In some embodiments, the user input includes a contact with a manipulation affordance on the content entry user interface. In some embodiments, upon termination of the contact (e.g., lift-off), the content entry user interface element is maintained at the final location of the contact. In some embodiments, the content entry user interface element snaps to predetermined locations on the display. For example, the predetermined locations include the bottom of the display, the left side of the display, the right side of the display, or the top of the display. In some embodiments, the content entry user interface element changes its visual appearance to conform to the new location. For example, while at the top or bottom of the display, the content entry user interface element is horizontal and while at the left or right of the display, the content entry user interface element is vertical.
The above-described manner of moving the content entry user interface element (e.g., by receiving a user input selecting the content entry user interface element and dragging it to a different location) simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by giving the user the ability to move the content entry user interface element to reveal previously obscured portions of the user interface, without requiring the user to perform additional inputs to scroll the user interface or dismiss the content entry user interface element), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in response to moving the content entry user interface element in accordance with the movement of the user input (2240), such as in
In some embodiments, in response to moving the content entry user interface element in accordance with the movement of the user input, in accordance with a determination that the final location of the content entry user interface element does not satisfy the one or more location criteria, the electronic device displays (2244) the content entry user interface element at a second size, different from the first size, wherein while the content entry user interface has the second size, the content entry user interface element includes a representation of the current content entry mode and the given set of options corresponding to the current content entry mode, such as in
In some embodiments, the user input includes a touch-down and a movement dragging the content entry user interface element. In some embodiments, miniature or simplified mode comprises displaying a representation of the currently active content entry tool without displaying the other content entry tools and without displaying the set of options that correspond to the active content entry mode. In some embodiments, the predetermined locations in the display that cause the content entry user interface element to be displayed in miniature mode include the corners of the display (e.g., top-left, top-right, bottom-left, and bottom-right corners). In some embodiments, while the content entry user interface element is in the “miniature” mode, selection of the content entry user interface element causes the content entry user interface element to return to its default (e.g., full sized) mode.
The above-described manner of changing the size of the content entry user interface element (e.g., based on the location of the content entry user interface element) quickly and efficiently provides the user with options for inputting content while minimizing obstruction of the user interface (e.g., by allowing the user to move the content entry user interface element and miniaturize the content entry user interface element), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically changing the content entry user interface element to a smaller size if the user requested to move the content entry user interface element to predetermined locations, without requiring the user to perform additional inputs to move and resize the content entry user interface element), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the electronic device includes a global setting for configuring the electronic device to accept or ignore respective handwritten input from an object (e.g., a global setting to enable or disable content insertion from a finger) other than a respective device (e.g., a stylus) while in the first content entry mode and the second content entry mode (2246), such as in
In some embodiments, enabling the global setting results in inputs from the finger being treated similar to inputs from a stylus (e.g., such as to insert handwritten inputs that are converted into font-based text or to insert drawings). In some embodiments, disabling the global setting results in inputs from the finger being treated as a navigation inputs, selection inputs, or any other input other than a content insertion input (e.g., swipe gestures are optionally treated as scrolling inputs, tap inputs are optionally treated as selection inputs, etc.).
In some embodiments, the content entry user interface element includes an option that is selectable to accept or ignore the respective handwritten input from the object other than the stylus while in the first content entry mode and the second content entry mode without regard to a state of the global setting (2248), such as toggle option 2122 in
The above-described manner of managing handwritten inputs from a finger (e.g., by providing a global setting that can be overridden by a selectable option on the palette) provides a quick and efficient way of overriding the default response to finger inputs, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by providing the user another method to insert content, without requiring the user to switch to using a stylus or perform additional inputs and navigate to a settings user interface to toggle the global setting), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the user interface includes a second content entry region, the first content entry region supports a first set of content options, and the second content entry region supports a second set, different from the first set, of content options (2250), such as text entry region 2128 and content entry region 2130 in
For example, the first content entry region is configured to only accept a particular font type or font size while the second content entry region is configured to accept any font type or font size. In such embodiments, the content entry user interface element for the first content entry region does not include an option for selecting font type or font size while the content entry user interface element for the second content entry region includes options for font type and font size, even when the device is in the first content entry mode when entering handwritten input into the first and second content entry regions.
In some embodiments, while the electronic device is in the first content entry mode (2252), the electronic device receives (2254) a user input directed to a respective content entry region, such as in
In some embodiments, in response to receiving the user input directed to the respective content entry region (2256), the electronic device displays (2258), in the user interface, the content entry user interface element, such as in
In some embodiments, in accordance with a determination that the respective content entry region is the first content entry region, the content entry user interface element includes the first set of options corresponding to the first set of content options (2260), such as in
In some embodiments, in accordance with a determination that the respective content entry region is the second content entry region, the content entry user interface element includes a third set of options, different from the first set of options, corresponding to the second set of content options (2262), such as in
For example, if the second content entry region is configured to allow a user to change font settings, then the content entry user interface element includes option(s) for changing font settings. Thus, in some embodiments, the options included in the content entry user interface element depend on the type of content entry region that the user is inputting content into (e.g., the content entry region that has focus or the content entry region that the user has most recently interacted with or is currently interacting with). In response to the user input directed to the respective content entry region, content is inserted into the respective content entry region in accordance with the user input. For example, in some embodiments, if the user begins drawing in the respective content entry region, the palette is displayed and representations of the user's drawing is displayed in the respective content entry region.
The above-described manner of configuring the options displayed on the content entry user interface element based on the content entry region (e.g., by displaying options that are supported by the content entry region and not displaying options that are not supported by the content entry region) quickly and efficiently provides the user with options that are supported (e.g., by automatically determining what options are supported by the respective content entry region and not displaying options that are not supported), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by not providing the user with options that are inoperable or not supported), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
In some embodiments, while displaying the user interface including the first content entry region, wherein the first content entry region supports content entry in the first content entry mode and the second content entry mode (e.g., the first content entry region accepts font-based text and drawings such that the first content entry region accepts inputs while in the text entry mode and drawing mode), the electronic device receives (2264) a user input directed to the first content entry region, such as in
In some embodiments, in response to the user input directed to the first content entry region, the electronic device displays (2266), in the user interface, the content entry user interface element, such as in
In some embodiments, the most recently used content entry tool is the global most recently used content entry tool (e.g., across any content entry region and/or across any application). In some embodiments, the most recently used content entry is the most recently used content entry tool for the first content entry region. In some embodiments, the content entry user interface element includes the set of options corresponding to the content entry tool that is selected. In some embodiments, if the content entry region does not include any font-based text, then the device is configured to operate in the most recently used content entry mode and the content entry user interface element includes the options corresponding to the most recently used content entry mode. For example, if the user previously selected a pencil tool for inserting a pencil styled drawing in a respective content entry region and then dismisses the content entry user interface element, then the next time the user causes display of the content entry user interface element (e.g., causes to be displayed, such as in response to detection of handwritten input directed to the respective content entry region), the pencil tool is automatically selected and the set of options in the content entry user interface element correspond to the pencil tool. In another example, if the user previously selected a marker tool for inserting a marker styled drawing in a first content entry region, dismisses the content entry user interface element, and then displays the content entry user interface element for a second content entry region, then the marker tool is automatically selected and the set of options in the content entry user interface element correspond to the marker tool. In a third example, if a user inserts font-based text in a respective content entry region (e.g., using a virtual keyboard, a physical keyboard, the text entry tool, or any other text insertion function), selects the highlighter tool, and then dismisses the content entry user interface element, then the next time the user displays the content entry user interface element, the text entry tool is automatically selected even though the previously selected tool was the highlighter tool, because the respective content entry region has font-based text. In some embodiments, the tool that is automatically selected when the content entry user interface element is displayed dictates the content entry mode in which the device is configured. For example, if the automatically selected tool is the text entry tool, then the device is configured to operate in the handwriting text entry mode. In another example, if the automatically selected tool is the pencil tool, then the device is configured to operate in the pencil content entry mode. In some embodiments, if the device was configured to operate in a different mode before receiving the user input, then in response to the user input, the device switches to operating in the mode based on the automatically selected tool.
The above-described manner of displaying options associated with the most recently used content entry tool (e.g., if the content entry region does not include any font-based font) quickly and efficiently provides the user with options that the user is most likely to use, which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically configuring the device in the content entry mode that the user has most recently used, without requiring the user to perform additional inputs to switch to the desired content entry mode), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, in accordance with a determination that the first content entry region includes font-based text, the content entry user interface element includes the first set of options corresponding to the first content entry mode (2270), such as in
In some embodiments, if the content entry region includes font-based text, then the device is configured (e.g., upon touchdown detected in the content entry region) to operate in the first content entry mode (e.g., text entry mode). In some embodiments, the content entry user interface element includes the first set of options corresponding to the text entry tool. In some embodiments, in response to the user input, because the automatically selected tool is the content entry tool, the device is configured to operate in a respective content entry mode.
The above-described manner of displaying options associated with the text entry tool (e.g., if the content entry region includes font-based font) quickly and efficiently provides the user with options that the user is most likely to use (e.g., the user is likely to enter text due to the content entry region already including font-based font), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by automatically configuring the device in text entry mode, without requiring the user to perform additional inputs to switch to the desired content entry mode), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, displaying the content entry user interface element (2272), such as in
In some embodiments, displaying the content entry user interface element includes: in accordance with a determination that the user interface is a user interface of a second application, different from the first application, including a second option in the first set of options without including the first option in the first set of options (2276), such as in
In some embodiments, the options that are displayed in the content entry user interface element for a respective application is customized by the developer of the respective application. For example, a developer is able to add or remove options and/or tools from the default or standard list of options and/or tools. In some embodiments, a developer is able to customize the tools for all content entry regions in the respective application or customize the tools for each content entry region in the respective application individually.
The above-described manner of displaying options based on the application (e.g., by displaying options that the respective application is configured to allow) quickly and efficiently provides the user with options that are supported by the respective application (e.g., without providing the user with options that are inoperable or unsupported), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient, which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency.
In some embodiments, the user interface is a user interface of a first application (2278), such as user interface 2100 in
In some embodiments, while displaying the user interface and the second user interface, the electronic device receives (2284) a user input, such as in
In some embodiments, in response to receiving the user input, in accordance with a determination that the user input is directed to the second content entry region, the electronic device displays (2290) the content entry user interface element at a second location, different from the first location, corresponding to the second application, such as in
For example, if the user interface of the second application is displayed on the left half of the display, then the content entry user interface element is displayed on the left half of the display and/or centered on the second application. In some embodiments, if the size of the user interface is not the full size of the display, the content entry user interface element is displayed with a size other than full sized and with a set of options other than the full set of options corresponding to the active content entry mode. For example, if the first user interface is 25% of the width of the display (e.g., the second user interface is 75% of the width of the display), then the content entry user interface element displayed for the first user interface is optionally smaller than full size (e.g., 25%, 33%, 50%, 66% of full size, etc.) and one or more options are not displayed in the content entry user interface element. In some embodiments, the one or more options that are not displayed in the content entry user interface element are displayed in a sub-menu that is displayed in response to selection of an option in the content entry user interface element (e.g., one or more options that cannot fit in the content entry user interface element are moved into a sub-menu that is accessible from the content entry user interface element).
The above-described manner of displaying the content entry user interface element (e.g., centered on the application with the content entry region that the user is entering content into) quickly and efficiently indicates to the which user interface the content entry user interface element is associated with (e.g., by placing the content entry user interface element closer to the relevant application and further away from the application into which the user is not inserting content), which simplifies the interaction between the user and the electronic device and enhances the operability of the electronic device and makes the user-device interface more efficient (e.g., by reducing erroneous inputs to the wrong user interface), which additionally reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiency, while reducing errors in the usage of the device.
It should be understood that the particular order in which the operations in
The operations in the information processing methods described above are, optionally, implemented by running one or more functional modules in an information processing apparatus such as general purpose processors (e.g., as described with respect to
As described above, one aspect of the present technology potentially involves the gathering and use of data available from specific and legitimate sources to facilitate the streaming of content from one electronic device to another. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data can include demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, data or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information, usage history, handwriting styles, etc.
The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used to automatically perform operations with respect to interacting with the electronic device using a stylus (e.g., recognition of handwriting as text). Accordingly, use of such personal information data enables users to enter fewer inputs to perform an action with respect to handwriting inputs. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For instance, handwriting styles may be used to generate suggested font-based text for the user.
The present disclosure contemplates that those entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and/or privacy practices. In particular, such entities would be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of users. Such information regarding the use of personal data should be prominent and easily accessible by users, and should be updated as the collection and/or use of data changes. Personal information from users should be collected for legitimate uses only. Further, such collection/sharing should occur only after receiving the consent of the users or other legitimate basis specified in applicable law. Additionally, such entities should consider taking any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices. In addition, policies and practices should be adapted for the particular types of personal information data being collected and/or accessed and adapted to applicable laws and standards, including jurisdiction-specific considerations that may serve to impose a higher standard. For instance, in the US, collection of or access to certain health data may be governed by federal and/or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); whereas health data in other countries may be subject to other regulations and policies and should be handled accordingly.
Despite the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and/or software elements can be provided to prevent or block access to such personal information data. For example, the user is able to configure one or more electronic devices to change the discovery or privacy settings of the electronic device. For example, the user can select a setting that only allows an electronic device to access certain of the user's handwriting entry history when providing autocomplete suggestions.
Moreover, it is the intent of the present disclosure that personal information data should be managed and handled in a way to minimize risks of unintentional or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting data once it is no longer needed. In addition, and when applicable, including in certain health related applications, data de-identification can be used to protect a user's privacy. De-identification may be facilitated, when appropriate, by removing identifiers, controlling the amount or specificity of data stored (e.g., collecting location data at city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and/or other methods such as differential privacy.
Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data. For example, handwriting can be recognized based on aggregated non-personal information data or a bare minimum amount of personal information, such as the handwriting being handled only on the user's device or other non-personal information.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best use the invention and various described embodiments with various modifications as are suited to the particular use contemplated.
This application is a U.S. National Phase application under 35 U.S.C. § 371 of International Application No. PCT/US2020/031727, filed May 6, 2020, which claims the benefit of U.S. Provisional Patent Application No. 62/843,976, filed May 6, 2019, U.S. Provisional Patent Application No. 62/859,413, filed Jun. 10, 2019, and U.S. Provisional Patent Application No. 63/020,496, filed May 5, 2020, the contents of which are hereby incorporated by reference in their entireties for all purposes.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2020/031727 | 5/6/2020 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2020/227445 | 11/12/2020 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5367453 | Capps | Nov 1994 | A |
| 6323846 | Westerman et al. | Nov 2001 | B1 |
| 6570557 | Westerman et al. | May 2003 | B1 |
| 6677932 | Westerman | Jan 2004 | B1 |
| 7259752 | Simmons | Aug 2007 | B1 |
| 7614008 | Ording | Nov 2009 | B2 |
| 7633076 | Huppi et al. | Dec 2009 | B2 |
| 7653883 | Hotelling et al. | Jan 2010 | B2 |
| 7657849 | Chaudhri et al. | Feb 2010 | B2 |
| 7663607 | Hotelling et al. | Feb 2010 | B2 |
| 7844914 | Andre et al. | Nov 2010 | B2 |
| 7957762 | Herz et al. | Jun 2011 | B2 |
| 8006002 | Kalayjian et al. | Aug 2011 | B2 |
| 8239784 | Hotelling et al. | Aug 2012 | B2 |
| 8279180 | Hotelling et al. | Oct 2012 | B2 |
| 8381135 | Hotelling et al. | Feb 2013 | B2 |
| 8479122 | Hotelling et al. | Jul 2013 | B2 |
| 9348458 | Hotelling et al. | May 2016 | B2 |
| 9933937 | Lemay et al. | Apr 2018 | B2 |
| 9959037 | Chaudhri et al. | May 2018 | B2 |
| 10168899 | Feiszli et al. | Jan 2019 | B1 |
| 11429274 | Missig et al. | Aug 2022 | B2 |
| 20020015024 | Westerman et al. | Feb 2002 | A1 |
| 20020107885 | Brooks et al. | Aug 2002 | A1 |
| 20030071850 | Geidl | Apr 2003 | A1 |
| 20030214539 | Iwema et al. | Nov 2003 | A1 |
| 20040070573 | Graham | Apr 2004 | A1 |
| 20040085301 | Furukawa | May 2004 | A1 |
| 20050190059 | Wehrenberg | Sep 2005 | A1 |
| 20060017692 | Wehrenberg et al. | Jan 2006 | A1 |
| 20060033724 | Chaudhri et al. | Feb 2006 | A1 |
| 20060071910 | Kim et al. | Apr 2006 | A1 |
| 20060092138 | Kim | May 2006 | A1 |
| 20060197753 | Hotelling | Sep 2006 | A1 |
| 20090161958 | Markiewicz et al. | Jun 2009 | A1 |
| 20100107099 | Frazier et al. | Apr 2010 | A1 |
| 20100293460 | Budelli | Nov 2010 | A1 |
| 20120032925 | Sekiya | Feb 2012 | A1 |
| 20120229471 | Takai | Sep 2012 | A1 |
| 20120263381 | Yoshida | Oct 2012 | A1 |
| 20130046544 | Kay et al. | Feb 2013 | A1 |
| 20140019855 | Kim et al. | Jan 2014 | A1 |
| 20140108004 | Sternby et al. | Apr 2014 | A1 |
| 20140108976 | Steiner et al. | Apr 2014 | A1 |
| 20140194162 | Tsudik | Jul 2014 | A1 |
| 20140219564 | Demiya | Aug 2014 | A1 |
| 20140245139 | Lee et al. | Aug 2014 | A1 |
| 20140253465 | Hicks et al. | Sep 2014 | A1 |
| 20150058718 | Kim et al. | Feb 2015 | A1 |
| 20150082217 | Tumwattana | Mar 2015 | A1 |
| 20150205398 | Le | Jul 2015 | A1 |
| 20150212692 | Hyun et al. | Jul 2015 | A1 |
| 20150221106 | Winnemoeller et al. | Aug 2015 | A1 |
| 20160098186 | Sugiura | Apr 2016 | A1 |
| 20160349897 | Ishikawa | Dec 2016 | A1 |
| 20170024178 | Jeong et al. | Jan 2017 | A1 |
| 20170091153 | Thimbleby | Mar 2017 | A1 |
| 20170109032 | Mélinand et al. | Apr 2017 | A1 |
| 20180121074 | Peron et al. | May 2018 | A1 |
| 20180129391 | Files et al. | May 2018 | A1 |
| 20180329589 | Sonnino et al. | Nov 2018 | A1 |
| 20180335932 | Ta | Nov 2018 | A1 |
| 20200356254 | Missig et al. | Nov 2020 | A1 |
| 20200371629 | Hauenstein et al. | Nov 2020 | A1 |
| 20200401796 | Chateigner | Dec 2020 | A1 |
| 20210349606 | Chang et al. | Nov 2021 | A1 |
| 20210349627 | Chang et al. | Nov 2021 | A1 |
| 20240004532 | Soli et al. | Jan 2024 | A1 |
| Number | Date | Country |
|---|---|---|
| 108845757 | Nov 2018 | CN |
| 2912540 | Sep 2015 | EP |
| 2003-296029 | Oct 2003 | JP |
| 2008-27082 | Feb 2008 | JP |
| 2012-238295 | Dec 2012 | JP |
| 2015-049901 | Mar 2015 | JP |
| 2015-56154 | Mar 2015 | JP |
| 10-2014-0073225 | Jun 2014 | KR |
| 10-2015-0026022 | Mar 2015 | KR |
| 201112040 | Apr 2011 | TW |
| 2010119603 | Oct 2010 | WO |
| 2013169849 | Nov 2013 | WO |
| 2014105276 | Jul 2014 | WO |
| 2016200586 | Dec 2016 | WO |
| 2020227445 | Nov 2020 | WO |
| Entry |
|---|
| Notice of Allowance received for U.S. Appl. No. 16/868,449, dated Apr. 14, 2022, 8 Pages. |
| Final Office Action received for U.S. Appl. No. 17/031,678, dated Jan. 10, 2022, 30 pages. |
| International Search Report received for PCT Patent Application No. PCT/US2020/031727, dated Oct. 8, 2020, 10 pages. |
| International Search Report received for PCT Patent Application No. PCT/US2021/031866, dated Nov. 8, 2021, 7 pages. |
| Non-Final Office Action received for U.S. Appl. No. 16/868,449, dated May 26, 2021, 30 pages. |
| Non-Final Office Action received for U.S. Appl. No. 17/031,678, dated Jul. 8, 2021, 26 pages. |
| Non-Final Office Action received for U.S. Appl. No. 17/031,844, dated Dec. 3, 2021, 29 pages. |
| Notice of Allowance received for U.S. Appl. No. 16/868,449, dated Nov. 3, 2021, 8 pages. |
| Adak et al., “Extraction of Doodles and Drawings from Manuscripts”, ICIAP, 17th International Conference, Naples, Italy, Dec. 10, 2013, pp. 515-520. |
| Fosseide et al., “Character Recognition in the Presence of Occluding Clutter”, Proceedings of SPIE, vol. 7247, Retrieved on Aug. 10, 2021, Jan. 18, 2009, 13 pages. |
| Final Office Action received for U.S. Appl. No. 17/031,844, dated Aug. 8, 2022, 33 pages. |
| Notice of Allowance received for U.S. Appl. No. 17/031,678, dated Sep. 15, 2022, 10 pages. |
| Notice of Allowance received for U.S. Appl. No. 17/031,678, dated Feb. 1, 2023, 8 pages. |
| Hou et al., “An Algorithm of Calligraphy Beautification Based on Improved Velocity and Width Model”, Computer Engineering and Social Media (CSCESM), 2015 Second International Conference on Computer Science, 2015, pp. 124-127. |
| Examiner's Answer to Appeal Brief received for U.S. Appl. No. 17/031,844, mailed on Dec. 18, 2023, 14 pages. |
| International Search Report received for PCT Application No. PCT/US2023/021718, mailed on Nov. 3, 2023, 7 pages. |
| Non-Final Office Action received for U.S. Appl. No. 18/315,251, mailed on Mar. 7, 2024, 28 pages. |
| Toshiba Corporation, “Microsoft Windows for Pen Computing users Guide”, Version A1, Nov. 9, 1994, pp. 23-27, 70-77 (Official Copy Only), See attached Communication 37 CFR § 1.98(a)(3). |
| Matsushita et al., “Effect of Text/Non-text Classification for Ink Search Employing String Recognition”, IEEE, 2012 10th IAPR International Workshop on Document Analysis Systems, May 7, 2012, pp. 230-234. |
| Tianxiao, Liu, “Overview of Handwriting Input Technology”, CNKI, China Invention and Patent, vol. 12, [retrieved on Jul. 31, 2024], 2016, 5 pages (1 page of English Abstract and 4 pages of Official Copy). |
| Number | Date | Country | |
|---|---|---|---|
| 20220197493 A1 | Jun 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| 63020496 | May 2020 | US | |
| 62859413 | Jun 2019 | US | |
| 62843976 | May 2019 | US |
