Patent Grant
9971500
1. Field of Technology
The embodiments herein generally relate to electronic devices with touch screen displays. More particularly, the embodiments relate to user interfaces and gestures on a touch screen display.
2. Background
As portable electronic devices become more compact and the functions of the devices increase, it is increasingly difficult to design a user interface that allows users to easily interact with the multifunctional devices. This challenge is particularly significant for handheld portable devices, which have much smaller screens than desktop or laptop computers. This situation is unfortunate because the user interface is the gateway through which users receive not only content but also responses to user actions or behaviors, including user attempts to access a device's features, tools, and functions.
To improve user interaction with portable electronic devices, some portable devices include more pushbuttons that are each associated with a particular functionality, assign multiple functions to each pushbutton, or use complex menu systems to allow users to access the various functionalities of the devices. These conventional user interfaces often result in complicated key sequences and menu hierarchies that must be memorized by the user.
An electronic device with a touch-sensitive surface and display can execute a messaging application. A method for displaying swipe options in the messaging application comprises displaying a list of information corresponding to email messages. A contact is detected on the touch-sensitive surface of the electronic device and continuous movement of the contact is detected in a first direction, without breaking contact with the touch-sensitive surface, at a location on the display corresponding to a message of the plurality of email messages. One or more parameters of the movement of the contact are determined and, in accordance with a determination that a parameter of the movement meets a first criterion, user interface elements representing actions to be taken on the email message are displayed. Parameters include distance, rate, and any combination thereof and the first and second criterion can be a threshold distance, a threshold rate, or any combination thereof. The user interface elements representing actions can be selected and the action corresponding to the selected user interface element is applied to the email message, subsequently received email messages of the email message, the email thread associated with the email message, or any combination thereof. The user interface elements can be displayed by visually sliding out from a side of the email message. In accordance with a determination that the parameter of the movement meets a second criterion different from the first criterion, a default action is performed on the email message.
A method for, within a messaging application, associating an email message or subsequently received email messages with notification rules comprises, in response to receiving an action to compose a message, displaying a composition interface in the messaging application. Within the composition interface, a selectable user interface element is displayed that, when selected, changes notification rules for replies to the message. For the selectable user interface element can be a symbol such as a bell. A contact on the touch-sensitive surface at a location on the display corresponding to the selectable user interface element within the composition interface is detected and the contact indicative of selection of the selectable user interface element. In response to detection of the selection, changing an appearance of the selectable user interface element within the composition interface to indicate the activation of a notification rule that a subsequently received message in a conversation thread corresponding to the message will trigger a conversation notification. For example, the changed appearance of the selectable user interface element is visually distinguished from its previous state (e.g., outline of a bell) in a graphical indication such as color, size, and location. In subsequently received messages, a notification rule can display a VIP indicator, indicative of selection of the selectable user interface element, adjacent to the subsequently received messages. In addition, another notification rule displays conversation notifications, including push notifications, when subsequently received messages are received.
A method for ignoring a message or subsequently received messages comprises displaying information corresponding to a plurality of messages in a messaging application. An input is received at a location on the display corresponding to a displayed message and the input indicates deletion of the displayed message. For example, the location can include a selectable user interface element associated with a trash option. In response to receiving the input, selectable user interface elements representing deletion actions to be taken on subsequent messages in a conversation thread corresponding to the displayed message are displayed proximate to the location of the displayed message. For example, the selectable user interface elements are displayed in a sheet. In another embodiment, a prompt is displayed in response to receiving the input and can include options for trashing the displayed message or displaying additional deletion actions. In response to selection of one of the plurality of selectable user interface elements, the displayed message is deleted and a deletion action is set for subsequently received messages in the conversation thread.
A method for displaying composition interfaces along with email messages comprises displaying a first composition interface in a messaging application where the first composition interface includes a first message. While displaying the first composition interface, an action is received to compose a second message. In response to receiving the action, a second composition interface is displayed, the first composition interface is displayed in a first portion and the second composition interface is displayed in a second portion. For example, the first and second portions are displayed below a list of information corresponding to a plurality of messages in the messaging application. Alternatively, the first and second composition interfaces are displayed in a third portion below the list of information corresponding to the plurality of messages in the messaging application.
A method for displaying composition interfaces along with email messages comprises in response to a selection of a conversation view in a messaging application that is configured to display a plurality of email messages in the conversation view the following steps. One step includes identifying in the plurality of email messages a conversation thread that comprises email messages from the plurality of email messages. Another step includes determining information common to the email messages in the conversation thread. The common information comprises a conversation title based at least in part on a subject of at least one or more email messages in the conversation thread. Next steps include displaying a header comprising the common information in a header portion of the conversation view, and displaying, visually distinguished from the header area, a message area that comprises two or more of the email messages in the conversation thread. Each of the two or more email messages comprises sender information and content information.
A method for displaying composition interfaces along with email messages comprises receiving a search query that comprises at least one term for searching email messages in a messaging application. In response to receiving the search query, a step of the method includes determining whether one or more email messages in the messaging application include the at least one search term. Another step includes determining, for each of the one or more email messages in the messaging application including the search term, a field where the search term is found within the email message. When determining the field, the search term can appear in one or more of a plurality of different, predetermined locations. The method further includes displaying one or more email message query result groups, each of which is based on a different predetermined location and comprises the one or more email messages in the messaging application including the search term in the corresponding predetermined location.
For a better understanding of the various described embodiments, reference should be made to the Description of Embodiments below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures.
The figures depict, and the detail description describes, various non-limiting embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.
Many electronic devices have graphical user interfaces that include a large number of content units such as electronic messages, photos, audio files, video files, text, hyperlinks, icons, control elements, and/or visual representations thereof. Many of these content units have associated metadata (e.g., time and/or date stamps, file sizes, author identification, status identification, and security settings), which may be informative to the user. To simplify a user's interaction with user interfaces containing large numbers of content units, electronic devices may display only a subset of all content units and/or associated metadata on a display at any one time. A user may need to access content units and/or associated metadata contained within a given user interface, but which are not displayed at that time.
Some methods for accessing additional content units and/or metadata associated with displayed content units require a sequence of user inputs that navigate in a menu system. For example, with these methods, a user may need to select a user interface object to display a menu. The displayed menu may or may not include a menu option for accessing additional content units and/or metadata associated with displayed content units. When the displayed menu does not include the desired menu option, the user needs to search through multiple menus and/or sub-menus to find a menu that has the correct menu option. After displaying the menu that has the desired menu option, the user needs to select the menu option to initiate gaining access to the desired content. Alternatively, an electronic device may, by default, continuously display larger numbers of content units and/or metadata associated with displayed content units. This clutters a display and can be distracting for users, which reduces the efficiency of the user when using the device.
In embodiments described below, improved methods for accessing additional content units present, but not displayed, in a graphical user interface are achieved by providing user interfaces that allow unlocking of a predefined navigation direction using simple user contacts on a touch-sensitive surface (e.g., where there is a favored navigation axis, but some additional, infrequently accessed, information such as metadata is available in response to navigation in a direction different from the favored navigation axis). In other embodiments described below, improved methods for accessing metadata associated with displayed content units are achieved by providing user interfaces that reveal hidden metadata upon detection of simple user inputs, related to traditional navigation inputs, on a touch-sensitive surface. These methods streamline to process of accessing additional user interface content units and/or metadata associated with displayed content units without overly cluttering the user interface display.
Exemplary Devices
Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
It will also be understood that, although the terms first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.
The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As used herein, the term “if” is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event],” depending on the context.
Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as PDA and/or music player functions. Exemplary embodiments of portable multifunction devices include, without limitation, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, Calif. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and/ortouch pads), are, optionally, used. It should also be understood that, in some embodiments, the device is not a portable communications device, but is a desktop computer with a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).
In the discussion that follows, an electronic device that includes a display and a touch-sensitive surface is described. It should be understood, however, that the electronic device optionally includes one or more other physical user-interface devices, such as a physical keyboard, a mouse and/or a joystick.
The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and/or a digital video player application.
The various applications that are executed on the device optionally use at least one common physical user-interface device, such as the touch-sensitive surface. One or more functions of the touch-sensitive surface as well as corresponding information displayed on the device are, optionally, adjusted and/or varied from one application to the next and/or within a respective application. In this way, a common physical architecture (such as the touch-sensitive surface) of the device optionally supports the variety of applications with user interfaces that are intuitive and transparent to the user.
Attention is now directed toward embodiments of portable devices with touch-sensitive displays.
As used in the specification and claims, the term “intensity” of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch sensitive surface, or to a substitute (proxy) for the force or pressure of a contact on the touch sensitive surface. The intensity of a contact has a range of values that includes at least four distinct values and more typically includes hundreds of distinct values (e.g., at least 256). Intensity of a contact is, optionally, determined (or measured) using various approaches and various sensors or combinations of sensors. For example, one or more force sensors underneath or adjacent to the touch-sensitive surface are, optionally, used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., a weighted average) to determine an estimated force of a contact. Similarly, a pressure-sensitive tip of a stylus is, optionally, used to determine a pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and/or changes thereto, the capacitance of the touch-sensitive surface proximate to the contact and/or changes thereto, and/or the resistance of the touch-sensitive surface proximate to the contact and/or changes thereto are, optionally, used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the substitute measurements for contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurements). In some implementations, the substitute measurements for contact force or pressure are converted to an estimated force or pressure and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure).
As used in the specification and claims, the term “tactile output” refers to physical displacement of a device relative to a previous position of the device, physical displacement of a component (e.g., a touch-sensitive surface) of a device relative to another component (e.g., housing) of the device, or displacement of the component relative to a center of mass of the device that will be detected by a user with the user's sense of touch. For example, in situations where the device or the component of the device is in contact with a surface of a user that is sensitive to touch (e.g., a finger, palm, or other part of a user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in physical characteristics of the device or the component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is, optionally, interpreted by the user as a “down click” or “up click” of a physical actuator button. In some cases, a user will feel a tactile sensation such as an “down click” or “up click” even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movements. As another example, movement of the touch-sensitive surface is, optionally, interpreted or sensed by the user as “roughness” of the touch-sensitive surface, even when there is no change in smoothness of the touch-sensitive surface. While such interpretations of touch by a user will be subject to the individualized sensory perceptions of the user, there are many sensory perceptions of touch that are common to a large majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., an “up click,” a “down click,” “roughness”), unless otherwise stated, the generated tactile output corresponds to physical displacement of the device or a component thereof that will generate the described sensory perception for a typical (or average) user.
It should be appreciated that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. The various components shown in
Memory 102 optionally includes high-speed random access memory and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as CPU 120 and the peripherals interface 118, is, optionally, controlled by memory controller 122.
Peripherals interface 118 can be used to couple input and output peripherals of the device to CPU 120 and memory 102. The one or more processors 120 run or execute various software programs and/or sets of instructions stored in memory 102 to perform various functions for device 100 and to process data. In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are, optionally, implemented on a single chip, such as chip 104. In some other embodiments, they are, optionally, implemented on separate chips.
RF (radio frequency) circuitry 108 receives and sends RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to/from electromagnetic signals and communicates with communications networks and other communications devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and so forth. RF circuitry 108 optionally communicates with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication. The wireless communication optionally uses any of a plurality of communications standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.1 la, IEEE 802.1 lb, IEEE 802.1 lg and/or IEEE 802.1 ln), voice over Internet Protocol (VoIP), Wi-MAX, a protocol for e-mail (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.
Audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100. Audio circuitry 110 receives audio data from peripherals interface 118, converts the audio data to an electrical signal, and transmits the electrical signal to speaker 111. Speaker 111 converts the electrical signal to human-audible sound waves. Audio circuitry 110 also receives electrical signals converted by microphone 113 from sound waves. Audio circuitry 110 converts the electrical signal to audio data and transmits the audio data to peripherals interface 118 for processing. Audio data is, optionally, retrieved from and/or transmitted to memory 102 and/or RF circuitry 108 by peripherals interface 118. In some embodiments, audio circuitry 110 also includes a headset jack (e.g., 212,
I/O subsystem 106 couples input/output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripherals interface 118. I/O subsystem 106 optionally includes display controller 156, optical sensor controller 158, intensity sensor controller 159, haptic feedback controller 161 and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive/send electrical signals from/to other input or control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate embodiments, input controller(s) 160 are, optionally, coupled to any (or none) of the following: a keyboard, infrared port, USB port, and a pointer device such as a mouse. The one or more buttons (e.g., 208,
Touch-sensitive display 112 provides an input interface and an output interface between the device and a user. Display controller 156 receives and/or sends electrical signals from/to touch screen 112. Touch screen 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively termed “graphics”). In some embodiments, some or all of the visual output corresponds to user-interface objects.
Touch screen 112 has a touch-sensitive surface, sensor or set of sensors that accepts input from the user based on haptic and/or tactile contact. Touch screen 112 and display controller 156 (along with any associated modules and/or sets of instructions in memory 102) detect contact (and any movement or breaking of the contact) on touch screen 112 and converts the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages or images) that are displayed on touch screen 112. In an exemplary embodiment, a point of contact between touch screen 112 and the user corresponds to a finger of the user.
Touch screen 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. Touch screen 112 and display controller 156 optionally detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 112. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, Calif.
Touch screen 112 optionally has a video resolution in excess of 100 dpi. In some embodiments, the touch screen has a video resolution of approximately 160 dpi. The user optionally makes contact with touch screen 112 using any suitable object or appendage, such as a stylus, a finger, and so forth. In some embodiments, the user interface is designed to work primarily with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen. In some embodiments, the device translates the rough finger-based input into a precise pointer/cursor position or command for performing the actions desired by the user.
In some embodiments, in addition to the touch screen, device 100 optionally includes a touchpad (not shown) for activating or deactivating particular functions. In some embodiments, the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output. The touchpad is, optionally, a touch-sensitive surface that is separate from touch screen 112 or an extension of the touch-sensitive surface formed by the touch screen.
Device 100 also includes power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)) and any other components associated with the generation, management and distribution of power in portable devices.
Device 100 optionally also includes one or more optical sensors 164.
Device 100 optionally also includes one or more contact intensity sensors 165.
Device 100 optionally also includes one or more proximity sensors 166.
Device 100 optionally also includes one or more tactile output generators 167.
Device 100 optionally also includes one or more accelerometers 168.
In some embodiments, the software components stored in memory 102 include operating system 126, communication module (or set of instructions) 128, contact/motion module (or set of instructions) 130, graphics module (or set of instructions) 132, text input module (or set of instructions) 134, Global Positioning System (GPS) module (or set of instructions) 135, and applications (or sets of instructions) 136. Furthermore, in some embodiments memory 102 stores device/global internal state 157, as shown in
Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.
Communication module 128 facilitates communication with other devices over one or more external ports 124 and also includes various software components for handling data received by RF circuitry 108 and/or external port 124. External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted for coupling directly to other devices or indirectly over a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to and/or compatible with the 30-pin connector used on iPod (trademark of Apple Inc.) devices.
Contact/motion module 130 optionally detects contact with touch screen 112 (in conjunction with display controller 156) and other touch sensitive devices (e.g., a touchpad or physical click wheel). Contact/motion module 130 includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining an intensity of the contact (e.g., the force or pressure of the contact or a substitute for the force or pressure of the contact), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact). Contact/motion module 130 receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, optionally includes determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations are, optionally, applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., “multitouch”/multiple finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detect contact on a touchpad.
In some embodiments, contact/motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has “clicked” on an icon). In some embodiments at least a subset of the intensity thresholds are determined in accordance with software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of particular physical actuators and can be adjusted without changing the physical hardware of device 100). For example, a mouse “click” threshold of a trackpad or touch screen display can be set to any of a large range of predefined thresholds values without changing the trackpad or touch screen display hardware. Additionally, in some implementations a user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and/or by adjusting a plurality of intensity thresholds at once with a system-level click “intensity” parameter).
Contact/motion module 130 optionally detects a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and/or intensities of detected contacts). Thus, a gesture is, optionally, detected by detecting a particular contact pattern. For example, detecting a tap gesture includes detecting a finger-down event followed by detecting a finger-up (lift off) event at the same position (or substantially the same position) as the finger-down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and subsequently followed by detecting a finger-up (lift off) event.
Graphics module 132 includes various known software components for rendering and displaying graphics on touch screen 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast or other visual property) of graphics that are displayed. As used herein, the term “graphics” includes any object that can be displayed to a user, including without limitation text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations and the like.
In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is, optionally, assigned a corresponding code. Graphics module 132 receives, from applications etc., one or more codes specifying graphics to be displayed along with, if necessary, coordinate data and other graphic property data, and then generates screen image data to output to display controller 156.
Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator(s) 167 to produce tactile outputs at one or more locations on device 100 in response to user interactions with device 100.
Text input module 134, which is, optionally, a component of graphics module 132, provides soft keyboards for entering text in various applications (e.g., contacts 137, e-mail 140, IM 141, browser 147, and any other application that needs text input).
GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to telephone 138 for use in location-based dialing, to camera 143 as picture/video metadata, and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map/navigation widgets).
Applications 136 optionally include the following modules (or sets of instructions), or a subset or superset thereof:
Examples of other applications 136 that are, optionally, stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.
In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, contacts module 137 are, optionally, used to manage an address book or contact list (e.g., stored in application internal state 192 of contacts module 137 in memory 102 or memory 370), including: adding name(s) to the address book; deleting name(s) from the address book; associating telephone number(s), email address(es), physical address(es) or other information with a name; associating an image with a name; categorizing and sorting names; providing telephone numbers or e-mail addresses to initiate and/or facilitate communications by telephone 138, video conference 139, e-mail 140, or IM 141; and so forth.
In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 are, optionally, used to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in address book 137, modify a telephone number that has been entered dial a respective telephone number, conduct a conversation and disconnect or hang up when the conversation is completed. As noted above, the wireless communication optionally uses any of a plurality of communications standards, protocols and technologies.
In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch screen 112, display controller 156, optical sensor 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, videoconferencing module 139 includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions.
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, e-mail client module 140 includes executable instructions to create, send, receive, and manage e-mail in response to user instructions. In conjunction with image management module 144, e-mail client module 140 makes it very easy to create and send e-mails with still or video images taken with camera module 143.
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the instant messaging module 141 includes executable instructions to enter a sequence of characters corresponding to an instant message, to modify previously entered characters, to transmit a respective instant message (for example, using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephony-based instant messages or using XMPP, SIMPLE, or IMPS for Internet-based instant messages), to receive instant messages and to view received instant messages. In some embodiments, transmitted and/or received instant messages optionally include graphics, photos, audio files, video files and/or other attachments as are supported in a MMS and/or an Enhanced Messaging Service (EMS). As used herein, “instant messaging” refers to both telephony-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS).
In conjunction with RF circuitry 108, touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, workout support module 142 includes executable instructions to create workouts (e.g., with time, distance, and/or calorie burning goals); communicate with workout sensors (sports devices); receive workout sensor data; calibrate sensors used to monitor a workout; select and play music for a workout; and display, store and transmit workout data.
In conjunction with touch screen 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions to capture still images or video (including a video stream) and store them into memory 102, modify characteristics of a still image or video, or delete a still image or video from memory 102.
In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images.
In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 includes executable instructions to browse the Internet in accordance with user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.
In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, e-mail client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to do lists, etc.) in accordance with user instructions.
In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget modules 149 are mini-applications that are, optionally, downloaded and used by a user (e.g., weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or created by the user (e.g., user-created widget 149-6). In some embodiments, a widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets).
In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, the widget creator module 150 are, optionally, used by a user to create widgets (e.g., turning a user-specified portion of a web page into a widget).
In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions to search for text, music, sound, image, video, and/or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.
In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present or otherwise play back videos (e.g., on touch screen 112 or on an external, connected display via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).
In conjunction with touch screen 112, display controller 156, contact module 130, graphics module 132, and text input module 134, notes module 153 includes executable instructions to create and manage notes, to do lists, and the like in accordance with user instructions.
In conjunction with RF circuitry 108, touch screen 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 are, optionally, used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions; data on stores and other points of interest at or near a particular location; and other location based data) in accordance with user instructions.
In conjunction with touch screen 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, e-mail client module 140, and browser module 147, online video module 155 includes instructions that allow the user to access, browse, receive (e.g., by streaming and/or download), play back (e.g., on the touch screen or on an external, connected display via external port 124), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 141, rather than e-mail client module 140, is used to send a link to a particular online video.
Each of the above identified modules and applications correspond to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules are, optionally, combined or otherwise re-arranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.
In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad. By using a touch screen and/or a touchpad as the primary input control device for operation of device 100, the number of physical input control devices (such as push buttons, dials, and the like) on device 100 is, optionally, reduced.
The predefined set of functions that are performed exclusively through a touch screen and/or a touchpad optionally include navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates device 100 to a main, home, or root menu from any user interface that is displayed on device 100. In such embodiments, a “menu button” is implemented using a touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touchpad.
Event sorter 170 receives event information and determines the application 136-1 and application view 191 of application 136-1 to which to deliver the event information. Event sorter 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates the current application view(s) displayed on touch sensitive display 112 when the application is active or executing. In some embodiments, device/global internal state 157 is used by event sorter 170 to determine which application(s) is (are) currently active, and application internal state 192 is used by event sorter 170 to determine application views 191 to which to deliver event information.
In some embodiments, application internal state 192 includes additional information, such as one or more of: resume information to be used when application 136-1 resumes execution, user interface state information that indicates information being displayed or that is ready for display by application 136-1, a state queue for enabling the user to go back to a prior state or view of application 136-1, and a redo/undo queue of previous actions taken by the user.
Event monitor 171 receives event information from peripherals interface 118. Event information includes information about a sub-event (e.g., a user touch on touch-sensitive display 112, as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I/O subsystem 106 or a sensor, such as proximity sensor 166, accelerometer(s) 168, and/or microphone 113 (through audio circuitry 110). Information that peripherals interface 118 receives from I/O subsystem 106 includes information from touch-sensitive display 112 or a touch-sensitive surface.
In some embodiments, event monitor 171 sends requests to the peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripheral interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and/or for more than a predetermined duration).
In some embodiments, event sorter 170 also includes a hit view determination module 172 and/or an active event recognizer determination module 173.
Hit view determination module 172 provides software procedures for determining where a sub-event has taken place within one or more views, when touch sensitive display 112 displays more than one view. Views are made up of controls and other elements that a user can see on the display.
Another aspect of the user interface associated with an application is a set of views, sometimes herein called application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of a respective application) in which a touch is detected optionally correspond to programmatic levels within a programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is, optionally, called the hit view, and the set of events that are recognized as proper inputs are, optionally, determined based, at least in part, on the hit view of the initial touch that begins a touch-based gesture.
Hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies a hit view as the lowest view in the hierarchy which should handle the sub-event. In most circumstances, the hit view is the lowest level view in which an initiating sub-event occurs (i.e., the first sub-event in the sequence of sub-events that form an event or potential event). Once the hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.
Active event recognizer determination module 173 determines which view or views within a view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of a sub-event are actively involved views, and therefore determines that all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if touch sub-events were entirely confined to the area associated with one particular view, views higher in the hierarchy would still remain as actively involved views.
Event dispatcher module 174 dispatches the event information to an event recognizer (e.g., event recognizer 180). In embodiments including active event recognizer determination module 173, event dispatcher module 174 delivers the event information to an event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores in an event queue the event information, which is retrieved by a respective event receiver module 182.
In some embodiments, operating system 126 includes event sorter 170. Alternatively, application 136-1 includes event sorter 170. In yet other embodiments, event sorter 170 is a stand-alone module, or a part of another module stored in memory 102, such as contact/motion module 130.
In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a respective view of the application's user interface. Each application view 191 of the application 136-1 includes one or more event recognizers 180. Typically, a respective application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a user interface kit (not shown) or a higher level object from which application 136-1 inherits methods and other properties. In some embodiments, a respective event handler 190 includes one or more of: data updater 176, object updater 177, GUI updater 178, and/or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views 191 includes one or more respective event handlers 190. Also, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in a respective application view 191.
A respective event recognizer 180 receives event information (e.g., event data 179) from event sorter 170, and identifies an event from the event information. Event recognizer 180 includes event receiver 182 and event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of: metadata 183, and event delivery instructions 188 (which optionally include sub-event delivery instructions).
Event receiver 182 receives event information from event sorter 170. The event information includes information about a sub-event, for example, a touch or a touch movement. Depending on the sub-event, the event information also includes additional information, such as location of the sub-event. When the sub-event concerns motion of a touch, the event information optionally also includes speed and direction of the sub-event. In some embodiments, events include rotation of the device from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation (also called device attitude) of the device.
Event comparator 184 compares the event information to predefined event or sub-event definitions and, based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definitions 186. Event definitions 186 contain definitions of events (e.g., predefined sequences of sub-events), for example, event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in an event 187 include, for example, touch begin, touch end, touch movement, touch cancellation, and multiple touching. In one example, the definition for event 1 (187-1) is a double tap on a displayed object. The double tap, for example, comprises a first touch (touch begin) on the displayed object for a predetermined phase, a first lift-off (touch end) for a predetermined phase, a second touch (touch begin) on the displayed object for a predetermined phase, and a second lift-off (touch end) for a predetermined phase. In another example, the definition for event 2 (187-2) is a dragging on a displayed object. The dragging, for example, comprises a touch (or contact) on the displayed object for a predetermined phase, a movement of the touch across touch-sensitive display 112, and lift-off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.
In some embodiments, event definition 187 includes a definition of an event for a respective user-interface object. In some embodiments, event comparator 184 performs a hit test to determine which user-interface object is associated with a sub-event. For example, in an application view in which three user-interface objects are displayed on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user-interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object triggering the hit test.
In some embodiments, the definition for a respective event 187 also includes delayed actions that delay delivery of the event information until after it has been determined whether the sequence of sub-events does or does not correspond to the event recognizer's event type.
When a respective event recognizer 180 determines that the series of sub-events do not match any of the events in event definitions 186, the respective event recognizer 180 enters an event impossible, event failed, or event ended state, after which it disregards subsequent sub-events of the touch-based gesture. In this situation, other event recognizers, if any, that remain active for the hit view continue to track and process sub-events of an ongoing touch-based gesture.
In some embodiments, a respective event recognizer 180 includes metadata 183 with configurable properties, flags, and/or lists that indicate how the event delivery system should perform sub-event delivery to actively involved event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate how event recognizers interact, or are enabled to interact, with one another. In some embodiments, metadata 183 includes configurable properties, flags, and/or lists that indicate whether sub-events are delivered to varying levels in the view or programmatic hierarchy.
In some embodiments, a respective event recognizer 180 activates event handler 190 associated with an event when one or more particular sub-events of an event are recognized. In some embodiments, a respective event recognizer 180 delivers event information associated with the event to event handler 190. Activating an event handler 190 is distinct from sending (and deferred sending) sub-events to a respective hit view. In some embodiments, event recognizer 180 throws a flag associated with the recognized event, and event handler 190 associated with the flag catches the flag and performs a predefined process.
In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver event information to event handlers associated with the series of sub-events or to actively involved views. Event handlers associated with the series of sub-events or with actively involved views receive the event information and perform a predetermined process.
In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates the telephone number used in contacts module 137, or stores a video file used in video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 176 creates a new user-interface object or updates the position of a user-interface object. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends it to graphics module 132 for display on a touch-sensitive display.
In some embodiments, event handler(s) 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of a respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.
It shall be understood that the foregoing discussion regarding event handling of user touches on touch-sensitive displays also applies to other forms of user inputs to operate multifunction devices 100 with input-devices, not all of which are initiated on touch screens. For example, mouse movement and mouse button presses, optionally coordinated with single or multiple keyboard presses or holds; contact movements such as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs; movement of the device; oral instructions; detected eye movements; biometric inputs; and/or any combination thereof are optionally utilized as inputs corresponding to sub-events which define an event to be recognized.
Device 100 optionally also includes one or more physical buttons, such as “home” or menu button 204. As described previously, menu button 204 is, optionally, used to navigate to any application 136 in a set of applications that are, optionally executed on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on touch screen 112.
In one embodiment, device 100 includes touch screen 112, menu button 204, push button 206 for powering the device on/off and locking the device, volume adjustment button(s) 208, Subscriber Identity Module (SIM) card slot 210, head set jack 212, and docking/charging external port 124. Push button 206 is, optionally, used to turn the power on/off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlock process. In an alternative embodiment, device 100 also accepts verbal input for activation or deactivation of some functions through microphone 113. Device 100 also, optionally, includes one or more contact intensity sensors 165 for detecting intensity of contacts on touch screen 112 and/or one or more tactile output generators 167 for generating tactile outputs for a user of device 100.
Each of the above identified elements in
Attention is now directed towards embodiments of user interfaces (“UI”) that is, optionally, implemented on portable multifunction device 100.
It should be noted that the icon labels illustrated in
Although some of the examples which follow will be given with reference to inputs on touch screen display 112 (where the touch sensitive surface and the display are combined), in some embodiments, the device detects inputs on a touch-sensitive surface that is separate from the display, as shown in
Additionally, while the following examples are given primarily with reference to finger inputs (e.g., finger contacts, tap gestures, finger swipe gestures), it should be understood that, in some embodiments, one or more of the finger inputs are replaced with input from another input device (e.g., a mouse based input or stylus input). For example, a swipe gesture is, optionally, replaced with a mouse click (e.g., instead of a contact) followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is, optionally, replaced with a mouse click while the cursor is located over the location of the tap gesture (e.g., instead of detection of the contact followed by ceasing to detect the contact). Similarly, when multiple user inputs are simultaneously detected, it should be understood that multiple computer mice are, optionally, used simultaneously, or a mouse and finger contacts are, optionally, used simultaneously.
As used herein, the term “focus selector” refers to an input element that indicates a current part of a user interface with which a user is interacting. In some implementations that include a cursor or other location marker, the cursor acts as a “focus selector,” so that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in
Attention is now directed towards embodiments of user interfaces (“UI”) and associated processes that may be implemented on an electronic device with a display and a touch-sensitive surface, such as device 300 or portable multifunction device 100.
As mentioned previously, the device 100 includes multiple applications 136 configured to execute on the device 100. One of the multiple applications 136 is the email client module 140 and a user can interact with the application on the computing device 100. As will be further described below, the email client module 140 has various user interfaces to enhance the user experience with the computing device 100 when using the application.
Swipe Options in an Email Client Module UI and Gestures
Furthermore,
In one embodiment, the launch gesture or any other suitable gesture herein includes a tap on an icon. In general, a gesture is considered to be “on” an icon if the user's finger(s) touch at least a portion of the icon displayed on the user interface 500. In
Responsive to user selection of the menu button, for example, through contact of the menu button 204, the device 100 dismisses the mail application view 504 of the “Mail” icon 418 from the user interface 500 and displays the icons as shown in
The mail application view 504 of the email application includes additional user interface elements such as a back element 518, an edit element 520, a refresh element 522, a new draft element 524, or any other suitable element that can be used in an email application. The back element 518, if selected, displays email accounts associated with the user or additional folders of an account. The edit element 520, if selected, displays options for a user to remove one or more email threads from the mail application view 504. The refresh element 522, if selected, refreshes the mail application view 504 and, for example, displays any new received emails. The new draft element 524, if selected, displays a new draft email in the email application.
In one embodiment, the display gesture includes dragging a contact (e.g., user's finger, a stylus) a threshold distance from an initial position to a second position on top of an email thread 506 displayed in the mail application view 504. For example, a contact 526 is detected by the device 100 at an initial position in the email thread 506B, as shown in
After completion of the display gesture or the threshold distance or rate is reached, user interface elements such as the swipe options 528 are displayed on one side (e.g., left, right) of the message portion 506B. In one embodiment, the swipe options 528 slide out continuously and/or proportionally to a distance or a normalized distance reached by the contact 526. Thus, if the contact 526 reached half the threshold distance, then half of the swipe options 528 are displayed on the right side of the message portion 506B, e.g., as shown by the swipe in progress of
In the embodiment where the swipe options 528 are partially displayed as the contact 526 is maintained, if the contact 526 is not maintained until the threshold distance is reached and/or broken, the swipe options 528 can “rubber band,” bounce back, or slide out of view to the original view of the message portion 506B, hiding the swipe options 528.
In the illustrated example, once the threshold distance is reached, the swipe options 528 are displayed on the right side of the message portion 506B and the message portion 506B is shifted in the direction of the swipe but adjacent to the displayed swipe options 528. For example, the swipe options 528 include a more option 528A, a notify option 528B, and a trash option 528C in one embodiment, as shown in
In one embodiment, the close gesture includes dragging a contact a threshold distance from an initial position to a second position on top of a message portion 506 displaying the swipe options 528 displayed in the mail application view 504. For example, a contact 526 is detected by the device 100 at an initial position in the message portion 506B, as shown in
After completion of the close gesture or the threshold distance or rate is reached, the swipe options 528 are no longer displayed. In one embodiment, the swipe options 528 slide in or close continuously and/or proportionally to the distance or a normalized distance reached by the contact 526, as previously described and as shown in
In the embodiment in which the swipe options 528 are partially hidden as the contact 526 is maintained, if the contact 526 is not maintained until the threshold distance is reached and/or broken, the swipe options 528 can “rubber band” or bounce back out and display the full swipe options 528 of the message portion 506B, as shown in
In addition to displaying swipe options 528, a default action can be associated with a gesture as well such as an expanded swipe option 628.
In one embodiment, the display gesture includes dragging a contact a threshold distance from an initial position to a second position on top of a message portion 506 displayed in the mail application view 504. For example, a contact 626 is detected by the device 100 at an initial position in the message portion 506B, as shown in
In one embodiment, the threshold distance to display an expanded swipe option 628 is greater than the threshold distance to display the swipe options described in
In another embodiment, one or more threshold rates can be used as indicators of a user request to display either the swipe options 528 or the expanded swipe option 628. For example, if the rate of the movement of the contact 626 is less than a first threshold rate, then the swipe options 528 are displayed. If the rate of the movement of the contact 626 is greater than a second threshold rate, then the expanded swipe option 628 is displayed. In one embodiment, the first threshold rate can be less than or equal to the second threshold rate.
After completion of the display gesture or the threshold distance or rate is reached, the expanded swipe option 628 is displayed across the width of the message portion 506B as shown in
In another embodiment, the expanded swipe option 628 does not slide out or is not displayed until the contact 626 reaches the threshold distance or second threshold distance from the preceding example or a threshold rate, and then snaps into place. In the embodiment where the expanded swipe option 628 is partially displayed as the contact 626 is maintained, if the contact 626 is not maintained until the threshold distance, such as the second threshold distance from the preceding example, is reached and/or broken, as shown in
In the illustrated example, once the threshold distance is reached, the expanded swipe option 628 is displayed instead of the message portion 506B and the message portion 506B is no longer shown in the email application. For example, the expanded swipe option 628 can be an option included in the swipe options 528 such as the trash option 528C or can be a different option that can be a default option or customized by the user of the device 100 through a settings application of the device 100, further described in conjunction with
In one embodiment, the close gesture includes dragging a contact a threshold distance from an initial position to a second position on top of a message portion 506 displayed in the mail application view 504. For example, a contact 626 is detected by the device 100 at an initial position in the message portion 506B, as shown in
In one embodiment, the threshold distance to close an expanded swipe option 628 is greater than the threshold distance to close the swipe options described in
In another embodiment, one or more threshold rates can be used as indicators of a user request to close either the swipe options 528 or the expanded swipe option 628. For example, if the rate of the movement of the contact 626 is greater than a first threshold rate, then the expanded swipe option 628 is closed and the message portion 506B is displayed. If the rate of the movement of the contact 626 is less than a second threshold rate, then the expanded swipe option 628 is closed or hidden and the swipe options 528 are displayed. Then, if the movement of the contact 626 continues for a longer distance or is repeated, then the swipe options 528 can also be closed or hidden and the message portion 506 displayed. In this example, the first threshold rate can be greater than or equal to the second threshold rate.
After completion of the close gesture or the threshold distance or rate is reached, the message portion 506B is displayed again. In one embodiment, the expanded swipe option 628 slides in or closes continuously and/or proportionally to the distance or normalized distance reached by the contact 626, as described in
In another embodiment, the expanded swipe option 628 does not slide in or is not closed until the contact 626 reaches the threshold distance, such as the first threshold distance from the preceding example. In the embodiment where the expanded swipe option 628 is partially hidden as the contact 626 is maintained, if the contact 626 is not maintained until the threshold distance or first threshold distance is reached and/or broken, the expanded swipe option 628 can “rubber band” or bounce back and be displayed instead of the message portion 506B, shown in
In the illustrated example, once the threshold distance to close the expanded swipe option 528 is reached or the close gesture is completed, the expanded swipe option 628 is hidden and the message portion 506B is shown in the email application, as shown in
In addition to displaying swipe options 528 and an expanded swipe option 628, additional one or more options can be displayed with a gesture on the other side of the message portion such as swipe option 728.
In one embodiment, the displayed or closed swipe option 728 is presented and animated in a similar manner as the swipe options 528 described in conjunction with
One or more of the previously mentioned options (e.g., options in the swipe options 528, the expanded swipe option 628, and the swipe option 728) can be selected through a gesture to present a sheet associated with the swipe option. Once selected, an action is performed on the associated email thread or message associated with the message portion, display a popup associated with the option, display a sheet associated with the option for further actions to take on the associated email thread, or any combination thereof. Selection of options is further described in conjunction with
Displaying Sheets for Options in an Email Client Module UI and Gestures
In one embodiment, the sheet 800 for the more option 528A, as shown in
Customization of Swipe Options in an Email Client Module UI
The options in the swipe options 528, the expanded swipe option 628 and the swipe option 728 can be customized by users in a settings application.
Method for Swipe Options in an Email Client Module UI
A list comprising information corresponding to a plurality of email messages is displayed 1005. The list includes portions of email messages of email threads, as described previously in conjunction with
A contact 526 on the touch-sensitive surface is detected 1010. Continuous movement of the contact 526 in a first direction, without breaking contact with the touch-sensitive surface, is detected 1015 at a location on the display corresponding to a message of the plurality of email messages, as shown previously in
One or more parameters of the movement of the contact are determined 1020. Parameters of the movement of the contact 526 include 1022 distance, rate, strength of pressure of the contact 526, number of contact points, or any other measurable parameter of the contact 526 and its movement, as described previously in
In accordance with a determination that a parameter of the movement meets a first criterion, a plurality of user interface elements representing actions to be taken on the email message is displayed 1025. User interface elements representing actions to be taken are selectable and include 1029 options displayed in the swipe options 528, the expanded swipe option 628, or swipe options 728. For example, the swipe options 528 include a more option 528A, a notify option 528B, and a trash option 528C. In one embodiment, the default action or expanded swipe option 628 is 1034 an option (e.g., trash option 528C) in the swipe options 528. The options displayed in the swipe options 528, the expanded swipe option 628, and swipe options 728 can be default actions or personalized by a user as described in conjunction with
In some embodiments, the plurality of user interface elements is displayed 1025 or visually slides out from a side (e.g., right) continuously and/or proportionally to a distance or a normalized distance reached by the contact 526 or based on the first criterion. Thus, as a result, the portion of the email message of the email thread displayed in the row associated with the plurality of user interface elements being displayed is slid off the display in the direction of the continuous movement and can be slid off simultaneously as the plurality of user interface elements slide into view on the row. In one embodiment, if the first criterion is not met, the plurality of user interface elements are not displayed 1025. In the embodiment in which the plurality of user interface elements slide out from a side continuously and/or proportionally to a distance or a normalized distance reached by the contact 526, if the first criterion is not met, the plurality of user interface elements rubber band back to the side and are not displayed. In some embodiments, the plurality of user interface elements collapse to a thin line along the bottom of the row displaying a portion of an email message of an email thread that also displays the plurality of user interface elements. In one embodiment, if contact is maintained by the user, whether the first criterion is met, for longer than a threshold duration, the plurality of user interface elements are not displayed, rubber band back and disappear, or collapse to a thin line.
In accordance with a determination that the parameter of the movement meets a second criterion different from the first criterion, a default action is performed 1030 on the email message. Criteria, such as the first criterion and second criterion, of parameters include various threshold distances (e.g., a threshold distance of the first criterion is less than a threshold distance of the second criterion), threshold rates, threshold strengths of pressure of the contact 526, threshold numbers of contact points, or any other suitable threshold for a parameter, as described previously. In some embodiments, the default action is not performed 1030 until a user selects the displayed default action, such as the expanded swipe option 628. For example, a user interface element associated with the default action is displayed across the entire row displaying a portion of an email message of an email thread 506. The user interface element, according to one embodiment, is the trash option 528C.
The default action can be displayed, in one embodiment, or slide out from the side (e.g., right, same as the side the plurality of user interface elements is displayed 1025) continuously and/or proportionally to a distance or a normalized distance reached by the contact 526, based on the second criterion, or only if the second criterion is met. In the embodiment in which the user interface element associated with the default action slides out from a side continuously and/or proportionally to a distance or a normalized distance reached by the contact 526, if the second criterion is not met, the plurality of user interface elements rubber band back to the displayed plurality of user interface elements (e.g., swipe options 528) or no options are displayed. In some embodiments, the user interface element associated with the default action collapses to a thin line along the bottom of the row displaying a portion of an email message of an email thread that also displays the plurality of user interface elements. In one embodiment, if contact is maintained by the user, whether the second criterion is met, for longer than a threshold duration, the user interface element is not displayed, rubber bands back to the previously displayed plurality of user interface elements, rubber bands and disappears, or collapses to a thin line.
The displayed user interface elements are selectable and, in some embodiments, upon selection of a user interface element, a sheet is displayed including additional user interface elements associated with additional actions to take on the message. If a user interface element associated with the default action (e.g., trash option 528C) is selected, a prompt is displayed requesting confirmation of the selection. Upon receiving confirmation of the selection of the user interface element associated with the default action, a sheet including additional user interface elements associated with additional actions to take on the message is also displayed.
VIP Threads in an Email Client Module UI and Gestures
An email thread can be assigned a higher priority than other email threads by a user through indicating the email thread as a VIP thread.
Future emails or email messages received in the email thread 1106A include a VIP thread indicator 1108 as shown in
In addition to the VIP thread indicator 1108, additional notification rules include a user associated with the account receiving push notifications for the email thread with the VIP thread indicator 1108, as shown in
VIP of Existing Threads in an Email Client Module UI and Gestures
Though previously described in the context of drafting a new email, existing email threads 1106, 506 can be associated with a VIP Thread indicator 1108 as well. As shown in
Responsive to selection of the turn off notifications action in the sheet 1300 in
Method for VIP Threads in an Email Client Module UI
Within the composition interface, a selectable user interface element is displayed 1510 that, when selected, changes notification rules for replies to the message. In one embodiment, the selectable user interface element displayed is 1512 a bell, an outline of a bell, a shaded bell, or any other suitable graphically altered bell. Alternative embodiments of the selectable user interface element are described further in conjunction with
Notification rules include 1514 displaying an additional user interface adjacent to subsequently received messages in the conversation thread corresponding to the message, receiving push notifications for subsequently received messages in the conversation thread corresponding to the message, and any combination thereof, as further described in conjunction with
A contact 1126 on the touch-sensitive surface is detected 1515 at a location on the display corresponding to the selectable user interface element within the composition interface. The contact 1126 is indicative of selection of the selectable user interface element. In response to detection of the selection, an appearance of the selectable user interface element is changed 1520 within the composition interface to indicate the activation of a notification rule that a subsequently received message in a conversation or email thread corresponding to the message will trigger a conversation notification, as described in conjunction with
In a similar method as described previously in conjunction with
Selection of the user interface element representing the action to change notification rules results in changed notification rules that are different or opposite of the notification rules described above. For example, the notification rules include not displaying an additional user element adjacent to subsequently received messages in the conversation thread corresponding to the message, not receiving push notifications for subsequently received messages in the conversation thread corresponding to the message, and any combination thereof. Alternatively, selection of the user interface element can display a sheet 1300 and 1400 for additional actions to change notification rules, as described in conjunction with
Ignoring Threads in an Email Client Module UI and Gestures
Email threads 506 also can be ignored. As shown in
Just as email threads can be ignored, email threads can also be un-ignored. In one embodiment, the email thread can be un-ignored manually (e.g., through the expanded sheet option 628) or automatically. For example, the email thread can be automatically un-ignored if a reengage condition is met such as a VIP sender replies to the email thread, the sender replies to the user only, the user's name is mentioned in an email message of the email thread, or any other user-specific information appears in the email thread.
Customization of Ignore Options in an Email Client Module UI
Method for Ignoring Threads in an Email Client Module UI
Input at a location on the display corresponding to a displayed message is received 1810. The input indicates deletion of the displayed message. Input, for example, is a contact 1626 with the location on the display corresponding to the displayed message. In addition, the location on the display corresponding to the displayed message can display a selectable user interface element corresponding to a request for deletion of the displayed message. In one embodiment, the selectable user interface element is an expanded swipe option 628 displayed instead of the displayed message, as shown and described in
In response to receiving the input, a plurality of selectable user interface elements representing deletion actions to be taken on subsequent messages in a conversation thread corresponding to the displayed message is displayed 1815 proximate to the location of the displayed message. The plurality of selectable user interface elements representing deletion actions to be taken can be displayed in a sheet 1606 as shown in
In one embodiment, in response to receiving the input, a prompt 1600 is displayed 1817, as shown in
In response to a reengage condition being met by a subsequently received message of the displayed message, the selected deletion action for the subsequently received message and additional subsequently received messages of the subsequently received message of the display message is removed 1822. Reengage conditions include a sender previously designated as a VIP sender replies to the email thread, the sender replies to a user associated with the electronic device, the sender replies to a user associated with the electronic device only, the user's name is mentioned in an email message of the conversation thread, and any combination thereof as previously described in conjunction with
Draft Emails UI
Although the first and second draft UIs 1900A and 1900B are displayed at a width narrower than width of the touchscreen 112 of the device 100, in alternative embodiments, the draft UIs 1900A and 1900B can be displayed at a full width and height equal to the width and height of the touchscreen 112. In addition, the draft UIs 1900 also are moveable when displayed on top of other draft UIs 1900 or the mail application view 504 according to one embodiment. A user can touch-and-drag the draft UIs 1900 through a contact with the header or a portion of the draft UIs 1900.
Draft UIs 1900 displayed on top of the mail application view 504 or the mail application view 504 and the draft section 1902 can be minimized to the draft section 1902 using a minimize gesture as shown in
In one embodiment, the minimize gesture includes dragging a contact a threshold distance from an initial position to a second position on top of the mail application view 504. For example, a contact 1926 is detected by the device 100 at an initial position in the message portion 506B, as shown in
After completion of the minimize gesture or the threshold distance or rate is reached, draft UIs 1900 displayed on top of the mail application view 504 are minimized or dropped down into a draft section 1902, as shown in
Draft UIs 1900 displayed in a draft section 1902 or on top of the mail application view 504 can be removed with a remove gesture as shown in
While a draft UI 1900B is displayed on top of the mail application view 504, the user can still access the email messages associated with message portions 506 in their account.
Illustrated in
In an alternative embodiment, in response to receiving a draft gesture through a contact 2326 in
In yet another embodiment, in response to receiving a draft gesture through a contact 2326 in
While a draft UI 1900B is displayed in a larger draft section 1902 or on top of the thread section 2204 and the draft section 1902, the user can still access the email threads 506, 2306 in their account.
In response to receiving 2810 the action, a second composition interface (e.g., 1900B in FIG. C) is displayed 2815, the first composition interface is displayed 2820 in a first portion and the second composition interface is displayed 2825 in a second portion. In addition, the first and the second composition interfaces are resizable. In various embodiments, the area of the first portion and the area of the second portion can be equal or different. For example, the area of the second portion is greater than the area of the first portion and this can be in response to the second portion displaying the second composition interface while displaying the first composition interface. Since the second composition interface was displayed while the first composition interface was displayed, the second composition interface can also be displayed on top of the first composition interface in various embodiments.
A second contact (e.g., 1926 and 2426) can be received on the touch-sensitive surface at a location corresponding to a composition interface (e.g., the first composition interface or the second composition interface). Continuous movement of the second contact in a first direction, without breaking the second contact with the touch-sensitive surface, is detected, as shown in
In a similar manner, a third contact can be received on the touch-sensitive surface at a location corresponding to the third portion. Continuous movement of the second contact in a second direction, without breaking the third contact with the touch-sensitive surface, is detected. One or more parameters of the movement of the third contact are determined and, in accordance with a determination that a parameter of the movement meets a first criterion, the first composition interface and the second composition interface are displayed in a fourth portion below the list comprising information corresponding to the plurality of email messages associated with the messaging application, wherein the area of the fourth portion greater than the area of the third portion, as shown in
Composition interfaces can also be removed. A fifth contact (e.g., 2026 and 2526) is received on the touch-sensitive surface at a location corresponding to first composition interface and continuous movement of the fifth contact in a third direction, without breaking the fifth contact with the touch-sensitive surface, is detected, as shown in
In one embodiment, the first direction is a downward motion, substantially parallel with the orientation of the display and the second direction is opposite of the first direction and still substantially parallel with the orientation of the display. The third direction, then, is substantially perpendicular to the first and second direction and, therefore, to the orientation of the display. The one or more parameters include distance, rate, strength of pressure of the contact, number of contact points, and any combination thereof, as previously described in conjunction with
Conversation View in an Email UI
Contact 2910 is detected by the device 100 on the email thread 506B. The contact 2910 in one embodiment is a tap on the email thread 506B. Responsive to the detected contact 2910, the application view 504 of the email application displays email messages 2920A, 2920B, 2920C, 2920D, and 2920E from the email thread 506B as illustrated in
As displayed, an additional user interface element that is a thread header element 2930 is displayed in the application view 504 above the displayed email message 2920A. The thread header element 2930 may show the subject line common to the email messages and the number of email messages (items) in the email thread 506B. Furthermore, the title element 2931 displays the character string “Thread” indicating to the user that an email thread is displayed in the application view 504.
Similar to the first two lines in the thread participant section 2934, only the first name of the senders may be displayed in any additional lines. In other embodiments, first and abbreviated or unabbreviated last name, or last name followed by comma and first name for each participant are displayed as described above. Senders who were not originally part of the email thread, but added by the user of the email application to the thread, as described below, have a “+” sign displayed in front of their displayed names 2941. In comparison, senders who are excluded by the user from the email thread, as described below, have their displayed names 2942 struck through as shown in
A deselect-sender gesture in the application view 504 associated with the email thread 506B includes a tap on a sender name in the thread participant section 2934 provided that the deselect-sender gestures follows the select-sender gesture for that particular sender name. Following the first expand gesture, alone or in combination with a second expand gesture, without any prior select-sender gesture all displayed names in the thread participant section 2934 are selected, and the deselect-sender gesture for these displayed names would have no effect, leaving all displayed names selected. In this case, selecting a displayed name with a select-sender gesture unselects all displayed and selected names that are not selected by the select-sender gesture. The unselected, displayed names as described above appear visually distinguished from the selected sender name. For the example shown in
As illustrated in
Search Result Groups in an Email UI
As illustrated in
Contact 3350 is detected by the device 100 on a search text input element 3360 in the search box 3320. The contact 3350 includes, but is not limited to, a tap on the input element 3350. Responsive to the detected contact 3350, a search string composition interface or draft UI 3370 is shown as illustrated in
Responsive to inputting a search text, the email application searches for matches within email messages of a selected mailbox. The mailbox is selected based on the select-mailbox option 3330. Email messages in the selected mailbox that contain matches to the search text are automatically separated into different search result groups based on the location of the search term in the email. The search result groups include query result groups based on the sender information, recipient information, subject matter, and message body, labelled “From,” “To/Cc,” “Subject,” and “Message,” respectively. Association of an email message containing a match with a search result group is based on whether the matched text is part of the sender information, recipient information, subject matter, or message body, respectively. For example, if the matched text is found within the sender information, the corresponding email message is associated the search result group based on sender information (e.g., “From”). Thus, each query result groups is representative of where the matched text is located in an email message. If the matched text occurs in multiple locations within an email message, e.g. the sender information and the message body, the email message is associated with each corresponding search result group. Other search result groups can be defined by the user, for example, based on date, attachment size, sender location or any other attribute or field available in the email message.
As illustrated in
Only up to three email messages that match the search text are initially displayed for each search result group in the application view 504. If more than three email messages are included in a search result group, a more option 3387, labelled for example “Show More Results,” is displayed at the bottom of the corresponding search result group in the application view 504. The more option 3387 allows for displaying additional email messages included in the corresponding search result group and not already displayed. In response to an option gesture including a tap (not shown), the more option 3387 displays those additional email messages within its search result group. The more option 3387 is similar to the more option 2940 described above with respect to
The disclosure herein has been described in particular detail with respect to one possible embodiment. Those of skill in the art will appreciate that other embodiments may be practiced. First, the particular naming of the components and variables, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. Also, the particular division of functionality between the various system components described herein is merely exemplary, and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead performed by a single component.
Some portions of above description present features in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally or logically, are understood to be implemented by computer programs. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules or by functional names, without loss of generality.
Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Certain aspects of the embodiments disclosed herein include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.
The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent to those of skill in the art, along with equivalent variations. In addition, the present invention is not described with reference to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references to specific languages are provided for invention of enablement and best mode of the present invention.
The embodiments disclosed herein are well suited to a wide variety of computer network systems over numerous topologies. Within this field, the configuration and management of large networks includes storage devices and computers that are communicatively coupled to dissimilar computers and storage devices over a network, such as the Internet.
Finally, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure herein is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.
The application claims the benefit of Provisional Application No. 62/006,211, filed on Jun. 1, 2014, the content of which is incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4734689 | Kurakake | Mar 1988 | A |
| 5347295 | Agulnick et al. | Sep 1994 | A |
| 5481278 | Shigematsu et al. | Jan 1996 | A |
| 5627567 | Davidson | May 1997 | A |
| 5689550 | Garson | Nov 1997 | A |
| 5943052 | Allen et al. | Aug 1999 | A |
| 5973676 | Kawakura | Oct 1999 | A |
| 6037937 | Beaton et al. | Mar 2000 | A |
| 6154758 | Chiang | Nov 2000 | A |
| 6185551 | Birrell | Feb 2001 | B1 |
| 6209100 | Robertson | Mar 2001 | B1 |
| 6259436 | Moon et al. | Jul 2001 | B1 |
| 6381579 | Gervais | Apr 2002 | B1 |
| 6492979 | Kent et al. | Dec 2002 | B1 |
| 6542171 | Satou et al. | Apr 2003 | B1 |
| 6922816 | Amin et al. | Jul 2005 | B1 |
| 6924822 | Card et al. | Aug 2005 | B2 |
| 6938220 | Shigematsu et al. | Aug 2005 | B1 |
| 6954899 | Anderson | Oct 2005 | B1 |
| 6995746 | Aymeric | Feb 2006 | B2 |
| 7007226 | Smith | Feb 2006 | B1 |
| 7007239 | Hawkins et al. | Feb 2006 | B1 |
| 7110864 | Restrepo et al. | Sep 2006 | B2 |
| 7222299 | Lim | May 2007 | B1 |
| 7437683 | Beezer et al. | Oct 2008 | B1 |
| 7489306 | Kolmykov-Zotov et al. | Feb 2009 | B2 |
| D611054 | Lin et al. | Mar 2010 | S |
| 7730401 | Gillespie et al. | Jun 2010 | B2 |
| 8131298 | Beyer, Jr. | Mar 2012 | B1 |
| 8289162 | Mooring et al. | Oct 2012 | B2 |
| 8330715 | Yadavalli et al. | Dec 2012 | B2 |
| 8332477 | Kaiserlian | Dec 2012 | B1 |
| 8341529 | Li et al. | Dec 2012 | B1 |
| 8347232 | Prud″Hommeaux et al. | Jan 2013 | B1 |
| 8358321 | Weidner | Jan 2013 | B1 |
| 8365059 | Walsh et al. | Jan 2013 | B2 |
| 8368723 | Gossweiler, III et al. | Feb 2013 | B1 |
| 8448083 | Migos et al. | May 2013 | B1 |
| 8520025 | Patterson et al. | Aug 2013 | B2 |
| 8665225 | Herz et al. | Mar 2014 | B2 |
| 8756522 | Lee et al. | Jun 2014 | B2 |
| 8842082 | Migos et al. | Sep 2014 | B2 |
| 8850350 | Bi et al. | Sep 2014 | B2 |
| 8863020 | Hymel | Oct 2014 | B2 |
| 9026932 | Dixon | May 2015 | B1 |
| 9032338 | Murrett et al. | May 2015 | B2 |
| 9047009 | King et al. | Jun 2015 | B2 |
| 9092132 | Migos et al. | Jul 2015 | B2 |
| 9104647 | Artin | Aug 2015 | B2 |
| 9113193 | Gardes et al. | Aug 2015 | B1 |
| 9117426 | Wieder | Aug 2015 | B2 |
| 9229634 | Herz et al. | Jan 2016 | B2 |
| 9250798 | Migos et al. | Feb 2016 | B2 |
| 9280263 | Kim | Mar 2016 | B2 |
| 9654426 | Underwood, IV et al. | May 2017 | B2 |
| 20020015064 | Robotham et al. | Feb 2002 | A1 |
| 20020051018 | Yeh | May 2002 | A1 |
| 20020180763 | Kung | Dec 2002 | A1 |
| 20030006967 | Pihlaja | Jan 2003 | A1 |
| 20030063073 | Geaghan et al. | Apr 2003 | A1 |
| 20030076306 | Zadesky | Apr 2003 | A1 |
| 20030135555 | Birrel | Jul 2003 | A1 |
| 20030142081 | Iizuka | Jul 2003 | A1 |
| 20030182310 | Charnock | Sep 2003 | A1 |
| 20040015548 | Lee | Jan 2004 | A1 |
| 20040017355 | Shim | Jan 2004 | A1 |
| 20040021643 | Hoshino et al. | Feb 2004 | A1 |
| 20040068409 | Tanaka et al. | Apr 2004 | A1 |
| 20040108995 | Hoshino | Jun 2004 | A1 |
| 20040140984 | Hinckley et al. | Jul 2004 | A1 |
| 20040178997 | Gillespie et al. | Sep 2004 | A1 |
| 20040218104 | Smith et al. | Nov 2004 | A1 |
| 20040263487 | Mayoraz et al. | Dec 2004 | A1 |
| 20040268265 | Berger | Dec 2004 | A1 |
| 20050144241 | Stata | Jun 2005 | A1 |
| 20050172001 | Zaner | Aug 2005 | A1 |
| 20050190970 | Griffin | Sep 2005 | A1 |
| 20050198143 | Moody | Sep 2005 | A1 |
| 20050223057 | Buchheit | Oct 2005 | A1 |
| 20050223058 | Buchheit | Oct 2005 | A1 |
| 20050223066 | Buchheit | Oct 2005 | A1 |
| 20050223067 | Buchheit | Oct 2005 | A1 |
| 20050223326 | Chang | Oct 2005 | A1 |
| 20050234910 | Buchheit | Oct 2005 | A1 |
| 20050262203 | Buchheit | Nov 2005 | A1 |
| 20050275637 | Hinckley et al. | Dec 2005 | A1 |
| 20060007174 | Shen | Jan 2006 | A1 |
| 20060022955 | Kennedy | Feb 2006 | A1 |
| 20060038796 | Hinckley et al. | Feb 2006 | A1 |
| 20060048072 | Jarrett et al. | Mar 2006 | A1 |
| 20060085757 | Andre et al. | Apr 2006 | A1 |
| 20060092142 | Gillespie et al. | May 2006 | A1 |
| 20060117273 | Smith | Jun 2006 | A1 |
| 20060125803 | Westerman et al. | Jun 2006 | A1 |
| 20060128404 | Klassen | Jun 2006 | A1 |
| 20060132456 | Anson | Jun 2006 | A1 |
| 20060190828 | Zaner | Aug 2006 | A1 |
| 20060199612 | Beyer, Jr. | Sep 2006 | A1 |
| 20060242607 | Hudson | Oct 2006 | A1 |
| 20060248151 | Belakovskiy | Nov 2006 | A1 |
| 20060253018 | Speier | Nov 2006 | A1 |
| 20060253418 | Charnock | Nov 2006 | A1 |
| 20060277488 | Cok et al. | Dec 2006 | A1 |
| 20060290679 | Lii | Dec 2006 | A1 |
| 20070060228 | Akasaka et al. | Mar 2007 | A1 |
| 20070062852 | Zachut et al. | Mar 2007 | A1 |
| 20070067744 | Lane et al. | Mar 2007 | A1 |
| 20070124503 | Ramos et al. | May 2007 | A1 |
| 20070150842 | Chaudhri et al. | Jun 2007 | A1 |
| 20070152978 | Kocienda et al. | Jul 2007 | A1 |
| 20070152980 | Kocienda et al. | Jul 2007 | A1 |
| 20070173267 | Klassen | Jul 2007 | A1 |
| 20070178941 | Fujimoto et al. | Aug 2007 | A1 |
| 20070192744 | Reponen | Aug 2007 | A1 |
| 20070195067 | Zotov et al. | Aug 2007 | A1 |
| 20070232360 | Bocking | Oct 2007 | A1 |
| 20070247442 | Andre et al. | Oct 2007 | A1 |
| 20070247446 | Orsley et al. | Oct 2007 | A1 |
| 20070268274 | Westerman | Nov 2007 | A1 |
| 20070277126 | Park et al. | Nov 2007 | A1 |
| 20070300182 | Bilow | Dec 2007 | A1 |
| 20080012835 | Rimon et al. | Jan 2008 | A1 |
| 20080036743 | Westerman et al. | Feb 2008 | A1 |
| 20080094356 | Ording et al. | Apr 2008 | A1 |
| 20080094368 | Ording et al. | Apr 2008 | A1 |
| 20080122796 | Jobs | May 2008 | A1 |
| 20080134101 | Newman | Jun 2008 | A1 |
| 20080153459 | Kansal | Jun 2008 | A1 |
| 20080165142 | Kocienda et al. | Jul 2008 | A1 |
| 20080165145 | Herz et al. | Jul 2008 | A1 |
| 20080165160 | Kocienda et al. | Jul 2008 | A1 |
| 20080168349 | Lamiraux et al. | Jul 2008 | A1 |
| 20080168404 | Ording | Jul 2008 | A1 |
| 20080180391 | Auciello et al. | Jul 2008 | A1 |
| 20080180408 | Forstall et al. | Jul 2008 | A1 |
| 20080229254 | Warner | Sep 2008 | A1 |
| 20080259040 | Ording et al. | Oct 2008 | A1 |
| 20080285587 | Balk | Nov 2008 | A1 |
| 20080294730 | Oral | Nov 2008 | A1 |
| 20080295001 | Moody | Nov 2008 | A1 |
| 20090015559 | Day et al. | Jan 2009 | A1 |
| 20090029674 | Brezina | Jan 2009 | A1 |
| 20090030872 | Brezina | Jan 2009 | A1 |
| 20090030919 | Brezina | Jan 2009 | A1 |
| 20090030933 | Brezina | Jan 2009 | A1 |
| 20090030940 | Brezina | Jan 2009 | A1 |
| 20090031244 | Brezina | Jan 2009 | A1 |
| 20090031245 | Brezina | Jan 2009 | A1 |
| 20090041052 | Balk | Feb 2009 | A1 |
| 20090047983 | Klassen | Feb 2009 | A1 |
| 20090052778 | Edgecomb et al. | Feb 2009 | A1 |
| 20090064047 | Shim et al. | Mar 2009 | A1 |
| 20090106415 | Brezina | Apr 2009 | A1 |
| 20090106676 | Brezina | Apr 2009 | A1 |
| 20090109182 | Fyke et al. | Apr 2009 | A1 |
| 20090122018 | Vymenets et al. | May 2009 | A1 |
| 20090125824 | Andrews et al. | May 2009 | A1 |
| 20090140986 | Karkkainen et al. | Jun 2009 | A1 |
| 20090144634 | Berger | Jun 2009 | A1 |
| 20090153288 | Hope et al. | Jun 2009 | A1 |
| 20090167700 | Westerman et al. | Jul 2009 | A1 |
| 20090176521 | Klassen | Jul 2009 | A1 |
| 20090189858 | Lev et al. | Jul 2009 | A1 |
| 20090228792 | Van Os et al. | Sep 2009 | A1 |
| 20090235196 | MacBeth et al. | Sep 2009 | A1 |
| 20090239587 | Negron | Sep 2009 | A1 |
| 20090249257 | Bove et al. | Oct 2009 | A1 |
| 20090249258 | Tang | Oct 2009 | A1 |
| 20090282360 | Park et al. | Nov 2009 | A1 |
| 20100011317 | Lee | Jan 2010 | A1 |
| 20100013780 | Ikeda | Jan 2010 | A1 |
| 20100057879 | Buchheit | Mar 2010 | A1 |
| 20100064017 | Buchheit | Mar 2010 | A1 |
| 20100064261 | Andrews et al. | Mar 2010 | A1 |
| 20100087172 | Klassen | Apr 2010 | A1 |
| 20100103124 | Kruzeniski et al. | Apr 2010 | A1 |
| 20100107050 | Wang et al. | Apr 2010 | A1 |
| 20100127972 | Yadavalli et al. | May 2010 | A1 |
| 20100156813 | Duarte et al. | Jun 2010 | A1 |
| 20100169766 | Duarte et al. | Jul 2010 | A1 |
| 20100188352 | Ikeda | Jul 2010 | A1 |
| 20100214237 | Echeverri et al. | Aug 2010 | A1 |
| 20100220062 | Antila | Sep 2010 | A1 |
| 20100235726 | Ording et al. | Sep 2010 | A1 |
| 20100235785 | Ording et al. | Sep 2010 | A1 |
| 20100235793 | Ording | Sep 2010 | A1 |
| 20100238139 | Goertz et al. | Sep 2010 | A1 |
| 20100241700 | Rasmussen | Sep 2010 | A1 |
| 20100241749 | Rasmussen | Sep 2010 | A1 |
| 20100250682 | Goldberg | Sep 2010 | A1 |
| 20100281397 | Buchheit | Nov 2010 | A1 |
| 20100283753 | Ohshita et al. | Nov 2010 | A1 |
| 20100293242 | Buchheit | Nov 2010 | A1 |
| 20100293460 | Budelli | Nov 2010 | A1 |
| 20100299599 | Shin et al. | Nov 2010 | A1 |
| 20100309148 | Fleizach et al. | Dec 2010 | A1 |
| 20110001694 | Homma et al. | Jan 2011 | A1 |
| 20110010182 | Turski | Jan 2011 | A1 |
| 20110010340 | Hung | Jan 2011 | A1 |
| 20110029918 | Yoo et al. | Feb 2011 | A1 |
| 20110037725 | Pryor | Feb 2011 | A1 |
| 20110039602 | McNamara et al. | Feb 2011 | A1 |
| 20110050607 | Park | Mar 2011 | A1 |
| 20110055753 | Horodezk et al. | Mar 2011 | A1 |
| 20110069006 | Liu et al. | Mar 2011 | A1 |
| 20110074699 | Marr et al. | Mar 2011 | A1 |
| 20110074710 | Weeldreyer | Mar 2011 | A1 |
| 20110078560 | Weeldreyer et al. | Mar 2011 | A1 |
| 20110106439 | Huang et al. | May 2011 | A1 |
| 20110122159 | Bergsten et al. | May 2011 | A1 |
| 20110128241 | Kang | Jun 2011 | A1 |
| 20110130182 | Namba | Jun 2011 | A1 |
| 20110175830 | Miyazawa | Jul 2011 | A1 |
| 20110179387 | Shaffer et al. | Jul 2011 | A1 |
| 20110191718 | Hinckley et al. | Aug 2011 | A1 |
| 20110191719 | Hinckley et al. | Aug 2011 | A1 |
| 20110199393 | Nurse et al. | Aug 2011 | A1 |
| 20110202877 | Lassonde et al. | Aug 2011 | A1 |
| 20110221684 | Rydenhag | Sep 2011 | A1 |
| 20110231499 | Stovicek | Sep 2011 | A1 |
| 20110231789 | Bukurak et al. | Sep 2011 | A1 |
| 20110231796 | Vigil | Sep 2011 | A1 |
| 20110258537 | Rives et al. | Oct 2011 | A1 |
| 20110264689 | Hirst | Oct 2011 | A1 |
| 20110302532 | Missig | Dec 2011 | A1 |
| 20110320978 | Horodezky et al. | Dec 2011 | A1 |
| 20120042278 | Vaisanen | Feb 2012 | A1 |
| 20120044177 | Ohta | Feb 2012 | A1 |
| 20120046947 | Fleizach | Feb 2012 | A1 |
| 20120088524 | Moldavsky | Apr 2012 | A1 |
| 20120096345 | Ho et al. | Apr 2012 | A1 |
| 20120110431 | Rosenfeld et al. | May 2012 | A1 |
| 20120120002 | Ota | May 2012 | A1 |
| 20120139844 | Ramstein et al. | Jun 2012 | A1 |
| 20120143972 | Malik | Jun 2012 | A1 |
| 20120166998 | Cotterill | Jun 2012 | A1 |
| 20120182226 | Tuli | Jul 2012 | A1 |
| 20120185498 | Loofbourrow | Jul 2012 | A1 |
| 20120192086 | Ghods | Jul 2012 | A1 |
| 20120210334 | Sutedja | Aug 2012 | A1 |
| 20120233571 | Wever | Sep 2012 | A1 |
| 20120311507 | Murrett et al. | Dec 2012 | A1 |
| 20130016129 | Gossweiler, III et al. | Jan 2013 | A1 |
| 20130024820 | Kirkpatrick | Jan 2013 | A1 |
| 20130038541 | Bakker | Feb 2013 | A1 |
| 20130050119 | Nemoto | Feb 2013 | A1 |
| 20130055119 | Luong | Feb 2013 | A1 |
| 20130076591 | Sirpal | Mar 2013 | A1 |
| 20130086480 | Sirpal | Apr 2013 | A1 |
| 20130093764 | Andersson | Apr 2013 | A1 |
| 20130097519 | Andersson | Apr 2013 | A1 |
| 20130097520 | Lewin | Apr 2013 | A1 |
| 20130097521 | Lewin | Apr 2013 | A1 |
| 20130097526 | Stovicek | Apr 2013 | A1 |
| 20130097534 | Lewin | Apr 2013 | A1 |
| 20130097539 | Mansson | Apr 2013 | A1 |
| 20130097562 | Kermoian | Apr 2013 | A1 |
| 20130113720 | Van Eerd et al. | May 2013 | A1 |
| 20130159893 | Lewin | Jun 2013 | A1 |
| 20130159902 | Kwak et al. | Jun 2013 | A1 |
| 20130167013 | Poliak | Jun 2013 | A1 |
| 20130185290 | Hirst | Jul 2013 | A1 |
| 20130214995 | Lewin | Aug 2013 | A1 |
| 20130222244 | Mak et al. | Aug 2013 | A1 |
| 20130227490 | Thorsander et al. | Aug 2013 | A1 |
| 20130263052 | Fong | Oct 2013 | A1 |
| 20130265225 | Nasiri et al. | Oct 2013 | A1 |
| 20130286435 | Anezaki et al. | Oct 2013 | A1 |
| 20130290291 | Loofbourrow | Oct 2013 | A1 |
| 20130311867 | Patterson et al. | Nov 2013 | A1 |
| 20140033032 | Reynolds et al. | Jan 2014 | A1 |
| 20140040835 | Hildreth et al. | Feb 2014 | A1 |
| 20140082521 | Carolan | Mar 2014 | A1 |
| 20140129457 | Peeler | May 2014 | A1 |
| 20140143728 | Coleman, Jr. et al. | May 2014 | A1 |
| 20140143738 | Underwood, IV et al. | May 2014 | A1 |
| 20140160073 | Matsuki | Jun 2014 | A1 |
| 20140191986 | Kim et al. | Jul 2014 | A1 |
| 20140210756 | Lee | Jul 2014 | A1 |
| 20140215382 | Engel | Jul 2014 | A1 |
| 20140222933 | Stovicek | Aug 2014 | A1 |
| 20140223381 | Huang et al. | Aug 2014 | A1 |
| 20140300543 | Kim | Oct 2014 | A1 |
| 20140309849 | Ricci | Oct 2014 | A1 |
| 20140351691 | Neil et al. | Nov 2014 | A1 |
| 20150028626 | Gopal | Jan 2015 | A1 |
| 20150067596 | Brown | Mar 2015 | A1 |
| 20150143258 | Carolan | May 2015 | A1 |
| 20150199371 | Loofbourrow | Jul 2015 | A1 |
| 20150212723 | Lim | Jul 2015 | A1 |
| 20150235370 | Abovitz et al. | Aug 2015 | A1 |
| 20150248233 | Murrett et al. | Sep 2015 | A1 |
| 20150250682 | Bangera | Sep 2015 | A1 |
| 20150286346 | Song et al. | Oct 2015 | A1 |
| 20150319297 | Beyer, Jr. | Nov 2015 | A1 |
| 20150346916 | Jisrawi et al. | Dec 2015 | A1 |
| 20150363066 | Lemay | Dec 2015 | A1 |
| 20160039426 | Ricci | Feb 2016 | A1 |
| 20160147438 | Migos et al. | May 2016 | A1 |
| 20160259528 | Foss | Sep 2016 | A1 |
| 20170083213 | Missig | Mar 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 101910988 | Dec 2010 | CN |
| 101943990 | Jan 2011 | CN |
| 0795811 | Sep 1997 | EP |
| 1271295 | Jan 2003 | EP |
| 2369820 | Sep 2011 | EP |
| 2381661 | Oct 2011 | EP |
| 2584770 | Apr 2013 | EP |
| 2770761 | Aug 2014 | EP |
| M401816 | Apr 2011 | TW |
| 199718508 | May 1997 | WO |
| 199938149 | Jul 1999 | WO |
| 200038042 | Jun 2000 | WO |
| 2002103669 | Dec 2002 | WO |
| 03067497 | Aug 2003 | WO |
| 2004070604 | Aug 2004 | WO |
| 2006020305 | Feb 2006 | WO |
| 2006115946 | Nov 2006 | WO |
| 2009072475 | Jun 2009 | WO |
| 2012178121 | Dec 2012 | WO |
| 2013016045 | Jan 2013 | WO |
| Entry |
|---|
| PCT Invitation to Pay Additional Fees and, Where Applicable, Protest Fee, PCT Application No. PCT/US2015/028626, dated Aug. 12, 2015, seven pages. |
| Taylor, C., “Mailbox Launches on Andriod,” Apr. 9, 2014, XP054975992, Retrieved from the Internet: URL:https://www.youtube.com/watch?v=sgL9883T97U [retrieved on Jul. 28, 2015], one page. |
| Office Action received for Taiwanese Patent Application No. 104116112, dated Apr. 19, 2017, 89 pages (31 pages of English translation and 58 pages of Official Copy). |
| Brodkin, Jon, “Review: In Windows 8.1, Mail is Finally Usable on Tablets and Desktops”, Ars Technica, Online available at <https://arstechnica.com/information-technology/2013/10/review-in-windows-8-1-mail-is-finally-usable-on-tablets-and-desktops/>, Oct. 17, 2013, 12 pages. |
| International Preliminary Report on Patentability Received for PCT Patent Application No. PCT/US2015/028626, dated Dec. 15, 2016, 31 pages. |
| International Search Report and Written Opinion Received for PCT Patent Application No. PCT/US2015/028626, dated Nov. 2, 2015, 37 pages. |
| Iyer, KS Sandhya, “Google Adds New Chat Box Style Compose and Reply Experience to Gmail”, NDTV Gadgets360.com, Online available at <http://gadgets.ndtv.com/internet/news/google-adds-new-chat-box-style-compose-and-reply-experience-to-gmail-286779, Oct. 31, 2012, 6 pages. |
| Chen et al., “Navigation Techniques for Dual-Display E-Book Readers”, CHI, Apr. 5-10, 2008, 10 pages. |
| Decision to Grant received for European Patent Application No. 11191132.7, dated Dec. 8, 2016, 3 pages. |
| Extended European Search Report received for European Patent Application No. 11191132.7, dated Jan. 24, 2012, 7 pages. |
| Final Office Action received for U.S. Appl. No. 11/961,700, dated Sep. 29, 2011, 33 pages. |
| Final Office Action received for U.S. Appl. No. 11/969,796, dated Sep. 12, 2011, 30 pages. |
| Final Office Action received for U.S. Appl. No. 12/217,038, dated Oct. 31, 2011, 83 pages. |
| Final Office Action received for U.S. Appl. No. 12/794,625, dated Aug. 1, 2013, 12 pages. |
| Final Office Action received for U.S. Appl. No. 12/794,625, dated Jul. 17, 2015, 13 pages. |
| Final Office Action received for U.S. Appl. No. 13/077,754 dated Jan. 14, 2015, 16 pages. |
| Final Office Action received for U.S. Appl. No. 13/217,747, dated Feb. 21, 2014, 22 pages. |
| Final Office Action received for U.S. Appl. No. 13/217,747, dated Sep. 10, 2014, 24 pages. |
| Final Office Action received for U.S. Appl. No. 14/292,262, dated Dec. 27, 2016, 3 Pages. |
| Final Office Action received for U.S. Appl. No. 14/707,250, dated Dec. 16, 2016, 16 pages. |
| Guimbretiere, Francois, “Paper Augmented Digital Documents”, ACM, vol. 5, No. 2, 2003, pp. 51-60. |
| Gutwin et al., “Improving Interpretation of Remote Gestures with Telepointer Traces”, In Proceedings of The 2002 Acm Conference on Computer Supported Cooperative Work, Nov. 16-20, 2002, pp. 49-57. |
| Hamilton, Dave, “iOS 7: Apple Mail's ‘Swipe to Delete’ Only Goes to the Left”, Available online at:—http://www.macobserver.com/tmo/article/ios-7-mails-swipe-to-delete-reverses-direction-from-ios-6, Sep. 23, 2013, 2 pages. |
| Intention to Grant received for European Patent Application No. 11191132.7, dated Jul. 22, 2016, 8 pages. |
| Intention to Grant received in European Patent Application No. 08727399.1, dated Nov. 30, 2012, 9 pages. |
| International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2015/027026, dated Dec. 15, 2016, 15 pages. |
| International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2007/089030, dated Jul. 7, 2009, 7 pages. |
| International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2008/050426, dated Jul. 7, 2009, 8 pages. |
| International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2009/039625, dated Jan. 5, 2011, 6 pages. |
| International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2012/039867, dated Dec. 12, 2013, 8 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2007/089030, dated Jul. 9, 2008, 9 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2009/039625, dated Jun. 5, 2009, 8 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2012/039867 dated Aug. 13, 2012, 11 pages. |
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2015/027026, dated Jun. 23, 2015, 17 Pages. |
| International Search Report and Written Opinion received in International Application No. PCT/US2008/050426, dated Jun. 13, 2008, 10 pages. |
| “Introduction to Word Processing”, available at <http://www2.raritanval.edu/departments/cis/full-time/Schwarz/security/word_one_pt2.htm>, Nov. 17, 2008, 9 pages. |
| Liao et al., “PACER: Fine-Grained Interactive Paper via Camera-touch Hybrid Gestures on a Cell Phone”, CHI 2010, ACM, Apr. 10-15, 2010, 10 pages. |
| Microsoft Corporation, “Microsoft Office Outlook 2003 Product Guide”, 2003, 73 pages. |
| “Microsoft Word 2000 Microsoft Corporation”, pages MSWord Figures 1-5, 2000. |
| Non-Final Office Action received for U.S. Appl. No. 13/077,711, dated Apr. 25, 2014, 15 pages. |
| Non-Final Office Action received for U.S. Appl. No. 13/077,754, dated Mar. 21, 2014, 13 pages. |
| Non-Final Office Action received for U.S. Appl. No. 11/961,700, dated Mar. 29, 2011, 26 pages. |
| Non-Final Office Action received for U.S. Appl. No. 11/969,796, dated Mar. 1, 2011, 28 pages. |
| Non-Final Office Action received for U.S. Appl. No. 12/217,038, dated Aug. 17, 2012, 42 pages. |
| Non-Final Office Action received for U.S. Appl. No. 12/217,038, dated Feb. 24, 2012, 167 pages. |
| Non-Final Office Action received for U.S. Appl. No. 12/217,038, dated Jun. 14, 2011, 92 pages. |
| Non-Final Office Action received for U.S. Appl. No. 12/217,038, dated Mar. 21, 2013, 46 pages. |
| Non-Final Office Action received for U.S. Appl. No. 12/794,625, dated Mar. 13, 2014, 13 pages. |
| Non-Final Office Action received for U.S. Appl. No. 12/794,625, dated Oct. 8, 2014, 12 pages. |
| Non-Final Office Action received for U.S. Appl. No. 12/794,625, dated Oct. 26, 2012, 7 pages. |
| Non-Final Office Action received for U.S. Appl. No. 13/077,754, dated Jul. 15, 2013, 11 pages. |
| Non-Final Office Action received for U.S. Appl. No. 13/077,754, dated Nov. 9, 2012, 9 pages. |
| Non-Final Office Action received for U.S. Appl. No. 13/217,747, dated Aug. 16, 2013, 19 pages. |
| Non-Final Office Action received for U.S. Appl. No. 13/217,747, dated Jan. 4, 2013, 17 pages. |
| Non-Final Office Action received for U.S. Appl. No. 14/197,111, dated Feb. 20, 2015, 60 pages. |
| Non-Final Office Action received for U.S. Appl. No. 14/292,262, dated Jul. 13, 2016, 17 Pages. |
| Non-Final Office Action received for U.S. Appl. No. 14/707,250, dated Jul. 15, 2016, 16 pages. |
| Non-Final Office Action received for U.S. Appl. No. 14/707,250, dated Jun. 26, 2017, 21 pages. |
| Non-Final Office Action received for U.S. Appl. No. 15/012,178, dated Sep. 8, 2017, 11 pages. |
| Notice of Allowance received for U.S. Appl. No. 13/077,754, dated Sep. 18, 2015, 6 pages. |
| Notice of Allowance received for U.S. Appl. No. 11/969,796, dated Apr. 29, 2013, 13 pages. |
| Notice of Allowance received for U.S. Appl. No. 12/217,038, dated Oct. 18, 2013, 15 pages. |
| Notice of Allowance received for U.S. Appl. No. 12/794,625, dated Sep. 7, 2016, 5 pages. |
| Notice of Allowance received for U.S. Appl. No. 13/077,711, dated Mar. 23, 2015, 8 pages. |
| Notice of Allowance received for U.S. Appl. No. 13/077,711, dated Nov. 28, 2014, 7 pages. |
| Notice of Allowance received for U.S. Appl. No. 13/217,747, dated Jan. 8, 2015, 11 pages. |
| Notice of Allowance received for U.S. Appl. No. 14/197,111, dated Aug. 25, 2015, 27 pages. |
| Notice of Allowance received for U.S. Appl. No. 14/292,262, dated Jun. 23, 2017, 6 Pages. |
| Notice of Allowance received for U.S. Appl. No. 14/292,262, dated Oct. 27, 2017, 5 pages. |
| Office Action received for Chinese Patent Application No. 201280026797.1, dated Nov. 4, 2015, 31 pages (18 pages of English Translation and 13 pages of Official Copy). |
| Office Action received for Chinese Patent Application No. 201280026797.1, dated Apr. 14, 2017, 18 pages (3 pages of English translation and 15 pages of Official Copy). |
| Office Action received for Chinese Patent Application No. 201280026797.1, dated Jul. 22, 2016, 19 paged (4 pages of English Translation and 15 pages of Official Copy). |
| Office Action received for European Patent Application No. 08727399.1, dated Dec. 7, 2009, 5 pages. |
| Office Action received for European Patent Application No. 08727399.1, dated Jun. 28, 2011, 4 pages. |
| Office Action received for European Patent Application No. 11191132.7, dated Jan. 14, 2015, 6 pages. |
| Office Action received for European Patent Application No. 12726302.8 dated Jan. 22, 2016, 6 pages. |
| Office Action received for Taiwanese Patent Application No. 104116112, dated Nov. 8, 2017, 12 pages (5 pages of English translation and 7 pages of Official Copy). |
| “The Textbook of the Future”, Nature, vol. 458, Apr. 2, 2009, pp. 568-570. |
| Villamore et al., “Touch Gesture Reference Guide”, available at < https://static.lukew.com/TouchGestureGuide.pdf>, Apr. 15, 2010, 7 pages. |
| Wang et al., “Detecting and Levaraging Finger Orientation for Interaction with Direct-Touch Surfaces”, ACM, Oct. 4-7, 2009, pp. 23-32. |
| Willems et al., “Pen Gestures in Online Map and Photograph Annotation Tasks”, Tenth International Workshop on Frontiers in Handwriting Recognition, Oct. 9, 2006, 7 pages. |
| Wilson, Andrew D., “ToughLight: An Imaging Touch Screen and Display for Gesture-Based Interaction”, ACM, Oct. 13-15, 2004, 8 pages. |
| Wu et al., “Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces”, Proceedings of the First IEEE International Workshop on Horizontal Interactive Human-Computer Systems, 2006, 8 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20150350143 A1 | Dec 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62006211 | Jun 2014 | US |
