Method and apparatus for displaying information during an instant messaging session

Information

  • Patent Grant
  • 10623347
  • Patent Number
    10,623,347
  • Date Filed
    Monday, July 8, 2019
    5 years ago
  • Date Issued
    Tuesday, April 14, 2020
    4 years ago
Abstract
A method and an apparatus are provided for controlling a graphical user interface to display information related to a communication session. Information relating to data produced by a first participant to the communication session is displayed on a first display unit, wherein the information produced by the first participant is displayed at a first position on the first display unit. Data is received from a second participant to the communication session, and information relating to the data received from the second participant is displayed on the first display unit, wherein the information received from the second participant is displayed at a second position on the first display unit. The first and second positions are horizontally spaced apart.
Description
BACKGROUND OF THE INVENTION
1. Field of the Invention

This invention relates generally to a user interface for displaying an exchange of messages during an instant messaging session, and, more particularly, to a method and apparatus for displaying instant message exchanges in a manner that graphically differentiates the participants in a conversation.


2. Description of the Related Art

Networks, such as the Internet, intranets, or other private or public networks, are ubiquitous. In fact, many computers are connected to one or more networks at the same time. For example, a business may have hundreds or even thousands of computers coupled to its own private network, which was, at least initially, used primarily for storage and exchange of computer files. At least some of these same business computers may also be coupled to the internet. Further, with the development of wireless devices, ad hoc networks may also be formed with properly configured portable devices. Even telephonic devices, such as cellular phones, pagers and the like, may be coupled to one or more of these networks. Small businesses and homes are also often connected in similar arrangements.


All of this connectivity has naturally led to communications between various users over these networks. For example, electronic mail (e-mail), because of its usefulness, is now commonplace. E-mail is now widely used by businesses and individuals, and in at least some instances has replaced more traditional forms of communications, such as mailed letters, facsimiles, telexes, and the like. However, e-mail has proven to be somewhat awkward when used to carry on an ongoing conversation.


Instant messaging, on the other hand, allows two or more users connected through these networks to carry on an interactive conversation. Exemplary instant messaging systems include Apple iChat, AOL Instant Messenger, Microsoft MSN Messenger, and the like. Typically, two or more users type in messages or select icons, which they send to one another. The receiving party(ies) may immediately respond with an appropriate message or icon. These instant messages are commonly all displayed in serial fashion, such as shown in FIG. 1, usually scrolling the user's screen from top to bottom. Commonly, each message is preceded by a label, such as BobbyD211 and Fred1432 in FIG. 1, indicating the identity of the author of the message. Heretofore, users have relied on these labels, or other limited indicia, to locate and identify messages from a particular party. Accordingly, it will be appreciated that the presentation of each message in substantially similar format makes it difficult to readily determine the authorship of one or more previous messages. Likewise, it is difficult to go back and quickly locate a previous message without reading through many previous messages.


The present invention is directed to overcoming or at least reducing one or more of the problems set forth above.


SUMMARY OF THE INVENTION

In one aspect of the present invention, a method is provided for displaying information related to a communication session. Information relating to data produced by a first participant to the communication session is displayed on a first display unit, wherein the information produced by the first participant is displayed at a first position on the first display unit. Data is received from a second participant to the communication session, and information relating to the data received from the second participant is displayed on the first display unit, wherein the information received from the second participant is displayed at a second position on the first display unit. The first and second positions are spatially distinct.


In another aspect of the present invention, a computer readable program storage device is provided and encoded with instructions that, when executed by a computer, performs a method. The method includes displaying information relating to data produced by a first participant to the communication session on a first display unit, wherein the information produced by the first participant is displayed at a first position on the first display unit. Data is received from a second participant to the communication session, and information relating to the data received from the second participant is displayed on the first display unit, wherein the information received from the second participant is displayed at a second position on the first display unit. The first and second positions are spatially distinct.


In still another aspect of the present invention, a graphical user interface for displaying information related to a communication session is provided. The interface is comprised of a first and a second spatially distinct region. The first region is adapted to display at least one message from a first participant to the instant messaging session. The second region is adapted to display at least one message from a second participant to the instant messaging session, and the first and second spatially distinct regions partially overlap and each include at least a portion that is free from overlapping.


In yet another aspect of the present invention, a method for displaying information related to a communication session is provided. Information relating to data produced by a participant to the communication session is received. The information received from the participant is then at least partially displayed within a speech balloon.


In still another aspect of the present invention, a graphical user interface for displaying information related to a communication session is provided. The graphical user interface comprises a first and second region, wherein the first region is adapted to display a speech balloon. The second region is adapted to display at least one message from a participant to the instant messaging session, wherein the second region is at least partially located within the first region.





BRIEF DESCRIPTION OF THE DRAWINGS

The invention may be understood by reference to the following description taken in conjunction with the accompanying drawings, in which like reference numerals identify like elements, and in which:



FIG. 1 illustrates a view of a screen representative of a graphical user interface of a prior art instant messaging system;



FIG. 2 illustrates a top-level diagram of one embodiment of a hardware system on which the present invention may be implemented;



FIG. 3 illustrates a flowchart of an embodiment of a graphical user interface that may be executed by components within the system of FIG. 1 to produce the exemplary screens of FIGS. 4 and 5;



FIG. 4 illustrates a first view of an exemplary screen representative of a graphical user interface;



FIGS. 5A-5B illustrate a second and third view of exemplary screens representative of a graphical user interface;



FIG. 6 illustrates a flowchart of an alternative embodiment of a graphical user interface that may be executed by components within the system of FIG. 1;



FIG. 7 illustrates a view of an exemplary screen representative of a graphical user interface;



FIG. 8 illustrates an alternative view of the exemplary screen of FIG. 7;



FIGS. 9A-9B illustrate speech bubbles; and



FIGS. 10A-10B illustrate speech balloons.





While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the description herein of specific embodiments is not intended to limit the invention to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims.


DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Illustrative embodiments of the invention are described below. In the interest of clarity, not all features of an actual implementation are described in this specification. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure.


Turning now to FIG. 2, a block diagram depicting a system 100 in accordance with embodiments of the present invention is illustrated. The system 100 includes a plurality of computing devices coupled together through one or more network connections. For example, a plurality of devices may be coupled together via a private or public network, such as a local area network (LAN) 102 or the Internet. The actual connection between the devices and the LAN 102 may take on one or more of any of a variety of forms, such as a network interface card (NIC), a modem, a digital subscriber line (DSL), a cable modem, a wireless connection, and the like. The devices coupled to the LAN 102 may include, for example, desktop computers, such as an Apple Macintosh® 104, a classic Apple Mac® 106, an IBM compatible personal computer (PC) 108, and the like. Further, these desktop computers, such as the Apple Macintosh® 104, may be coupled together via a smaller sub-LAN 110, with the sub-LAN 110 being coupled to the LAN 102. Portable devices, such as the Apple PowerBook® or iBook® 112, may also be coupled to the LAN 102, either directly or as part of the sub-LAN 110. Further, other consumer devices, such as cell phones, personal data assistants (PDAs), network appliances, and other embedded devices may be connected to the LAN 102 so as to employ aspects of the instant invention.


While the invention has been illustrated herein as being useful in a network environment, it also has application in other connected environments. For example, two or more of the devices described above may be coupled together via device-to-device connections, such as by hard cabling, radio frequency signals (e.g., 802.11(a), 802.11(b), 802.11(g), Bluetooth, or the like), infrared coupling, telephone lines and modems, or the like. The instant invention may have application in any environment where two or more users are interconnected and capable of communicating with one another.


Those skilled in the art will appreciate that network connections may include a variety of other equipment, such as routers, switches, telephone modems, wireless devices, cable modems, digital subscriber lines, and the like. This type of equipment is not illustrated or discussed in detail herein so as to avoid unnecessarily obfuscating the instant invention. For purposes of understanding the instant invention, it is sufficient to recognize that additional conventional equipment of this type may be useful in establishing and maintaining communications between the various users.


At least two of the devices in the system 100 have software, such as an application program, installed thereon to allow an instant messaging session to be initiated and conducted. An instant messaging session may include real-time or near real-time communications. FIG. 3 illustrates a flowchart of a portion of the software associated with initiating the instant messaging session and controlling a graphical user interface (GUI) used by the participants to the instant messaging session. In particular, the process begins at block 300 in a conventional manner with one of the two parties sending the other party an invitation to initiate an instant messaging session. Assuming that the other party accepts the invitation, the software on each party's computer initiates the GUI, which opens a window where both parties' messages and other pertinent information and controls are displayed. An exemplary representation of the GUI is shown in FIG. 4 and may be referenced simultaneous with the discussion of FIG. 3 herein for a more complete understanding of the operation of the instant invention.


The messages exchanged by the participants may contain information regarding an icon to be used to represent each party. For example, party A may select an icon, such as “Mary” 400 as a graphical representation of party A. Party B may receive and store the icon and then display it adjacent a message delivered by party A. The icon makes it easier for party B to more quickly identify those messages associated with party A. An exemplary exchange of messages in which party A has selected the icon “Mary” 400 and party B has selected the icon “Sue” 402 is shown in FIG. 4. Displaying unique graphical icons allows a user to readily identify the speaker with a quick glance. Additionally, displaying the icons adjacent each party's message allows the users to identify the speaker without looking away from the message region of the GUI. In an alternative embodiment, the user may elect to display not only the icon, but also the name associated with the author of the message. On the other hand, the user may elect to display only the name associated with the author of the message, preventing the icon from being displayed altogether, if desired.


Those skilled in the art will appreciate that the icons need not be delivered with each message. That is, party A may send an icon during the initial portion of the session, and party B will associate the icon with party A, store it locally, and then retrieve and display it each time a message is received from party A. Additionally, party A's icon may be overridden locally by party B. That is, party B may elect to display a different icon adjacent party A's messages, at least on the GUI viewed by party B. Party B may select any of a plurality of icons stored locally, and indicate through the local GUI, such as by pointing and clicking on various pull-down menus provided by the local GUI, that the selected icon should be used when displaying party A's messages.


The GUI may also use additional strategies to graphically differentiate the parties of the instant messaging session. For example, a sending party may send an indication of a color scheme in which his/her messages should be displayed. The receiving party may then, at his/her discretion, display the messages from the sender in the requested color scheme.


Alternatively, the receiving party may elect to override the sending parties requested preference, and instead display each party's message in its own distinct color. That is, party A, during an initialization phase, may indicate through the local GUI that any message received from party B should be displayed with red letters and a white background, and that any messages generated by himself, should be displayed with a yellow background and black letters. In either case, the color distinction allows the party to visually determine the author of a message without the need to read and understand an identifying name, such as is illustrated in the prior art at FIG. 1 (e.g., BobbyD211).


Allowing the sender to select the color and style, however, may lead to some confusion in the event that another participant to the instant messaging sessions elects a similar style and/or font. Empowering the receiver of the message to override locally the style and color choices indicated by the sender may help to alleviate any confusion. That is, the receiving party may elect to display the message with a different color and style than indicated by the sending party, at least on the GUI viewed by the receiving party. The receiving party may select any of a plurality of colors and styles stored locally, and indicate through the local GUI, such as by pointing and clicking on various pull-down menus provided by the local GUI, that the selected color and style should be used when displaying the received messages. Alternatively, the GUI may be programmed to automatically assign a different color to each participant.


An additional graphical distinction may be accomplished by partitioning the GUI into spatially distinct regions and then directing the messages to a region based upon its authorship. For example, the exemplary GUI of FIG. 4 has been generally divided into two horizontal regions, a left region 404 and a right region 406. For example, all messages generated by the local user (party A), represented by Mary 400, are displayed in the right region 406, and all messages generated by the remote user (party B), represented by Sue 402, are displayed in the left region 404. It should be appreciated that the assignment of left and right regions to parties A and B, respectively, may be reversed without departing from the spirit and scope of the instant invention. Moreover, it is anticipated that the various spatially distinct regions may overlap. That is, a message generated by Mary 400 may extend from the right region 406 and at least partially into the left region 404. Similarly, a message generated by Sue 402 may extend from the left region 404 and at least partially into the right region 406. Thus, the messages may at least partially overlap, depending on the length of the messages.


Further, depending upon the number of participants, it may be useful to define more than two spatially distinct regions. For example, where three participants are present, it may be useful to provide three horizontal regions.


In the exemplary GUI of FIG. 4, the text of the messages associated with Sue 402 is displayed in the left region and is left justified. Similarly the text of the messages associated with Mary 400 is displayed in the right region and is right justified. Those skilled in the art will appreciate that other justification schemes may be used without departing from the spirit and scope of the instant invention.


In one embodiment of the instant invention, the order in which the messages appear on the GUI generally corresponds to the order in which they were received. For example, in the illustrated embodiment of FIG. 4, each message is displayed below previously received messages so that the order of the conversation is preserved, with older messages appearing nearer the top of the GUI and newer messages appearing nearer the bottom of the GUI. As the display region of the GUI fills, old messages are scrolled up and out of view. A user may, however, activate a scrollbar mechanism 408 using conventional point and click techniques to alter the portion of the conversation presented in the GUI. For example, the user may move the scrollbar mechanism 408 upward to view an older portion of the conversation, or downward to view a more recent portion of the conversation.


To further enhance the readability and to provide further graphical identification of the author of each message appearing in the GUI, each message may be displayed in a speech balloon 410. The balloon 410 includes a tail section 412, which generally extends toward the icon associated with the author of the message. For example, each message from the user identified by the icon Mary 400 appears in a balloon 410 that has its tail section 412 extending generally toward the icon Mary 400. In the event that an icon is not associated with the author of the message, the tail section 412 is still useful to graphically illustrate the author. That is, since the GUI is divided into left and right horizontal regions, 404, 406 a speech balloon 410 located in the left horizontal region 404 with its tail section 412 extending toward the left will still provide a graphical indication of the author (e.g., Sue 402 in the embodiment of FIG. 4).


The size of the balloon 410 is controlled according to the length of the message. That is, the GUI receives a message, determines the length of the message, determines the size (e.g., based on the number of lines of text to be displayed) of the balloon 410 required to display the message, and then draws the balloon 410 with text in the appropriate horizontal portion of the GUI using the colors, style, and icon associated with the author of the message. A more detailed discussion of the sizing aspect of the speech balloons may be found below in conjunction with FIGS. 9 and 10.


During an instant messaging session it is often useful to indicate when a remote party is preparing a message to be sent. For example, after party A sends a message requesting a response, it is useful to know if party B is preparing the requested response. Knowing that the other party is about to respond allows a more natural flow to the conversation. For example, if party B does not answer a question from party A in a timely manner, party A may send a second, related request. Party B, however, may promptly respond to the first request, leaving party A to guess at whether the response applies to the first request, the second request, or both.


Accordingly, in the embodiment of the GUI shown in FIG. 3, at block 302 the software determines whether a message is being generated, and in the event that a message is being prepared, the software at block 304 delivers a signal to the other party indicating that a message is being generated. One method for determining whether a message is being generated is for the remote terminal to monitor an input field for any characters having been entered, such as via the keyboard, and report back to the local terminal. For example, if the software detects that a keystroke has been entered or that emoticons have been selected, then it assumes that a message is being prepared to be sent to the other party, and the software sends a signal indicating that a possible responsive message is being prepared.


At block 306, the software checks for a signal from the other remote party indicating that a message is being prepared. If such a signal is received, control transfers to block 308 where the GUI is activated to produce a graphical indication that a message is being prepared by the other party. An exemplary representation of the graphical indicator is shown in the exemplary GUI of FIG. 5. For example, a “thought bubble,” such as is often used in comic strips to indicate that a character is thinking, is displayed along with the icon associated with the party who is preparing the message. In the exemplary embodiment of FIG. 5, a “thought bubble” 500 provides a graphical indication that Sue 402 is currently preparing a message. For a number of reasons, the thought bubble 500 is particularly efficient for conveying the idea that the other party is preparing a response. First, the thought bubble 500 appears in the GUI in the same general region that a message would be displayed. Second, thought bubbles are common graphical representations familiar to many users. Third, because the thought bubble 500 is graphically similar to, but easily distinguishable from, the speech balloon 410, the user may intuitively understand its function as a precursor to an actual message. Accordingly, even inexperienced users may readily understand the function and operation of the instant messaging system, and will be able to more quickly participate in an instant messaging session at a higher level of proficiency.


In an alternative embodiment, incomplete or partial messages are communicated to the recipient as an indication that a message is being prepared. In this alternative embodiment, the partial message is accompanied by a graphic indication that the message is not yet complete, such as by “ . . . .” The partial messages are then periodically updated as more of the message is produced by the sender.


At block 310, the software checks to determine if a message has been received from the other party. If so, control transfers to block 312 where the software displays the text message (or emoticon, or the like) along with the icon associated with the author. In this instance, any corresponding thought bubble is replaced by the corresponding speech balloon and its accompanying text. In the illustrated embodiments of FIGS. 4 and 5, the messages received from the other, remote party are displayed on the left side 404 of a display window in the GUI. Additionally, the text message is presented in a speech balloon 410 and is left justified to further enhance its association with the other, remote party.


At block 314, the software checks to determine if the message being prepared by the local party is complete. If so, control transfers to block 316 and the software delivers the message over the network connection to the other party. The message is then displayed in the speech balloon 410 in replacement of the thought bubble. Additionally, the software displays the text message (or emoticon, or the like) along with the icon associated with the author in the local GUI. In the illustrated embodiments of FIGS. 4 and 5, the messages produced by the local party are displayed on the right side 406 of a display window in the GUI. Additionally, the text message is presented in a speech balloon 410 and is right justified to further enhance its association with the local party.


Those skilled in the art will appreciate that while the instant invention has been depicted in exemplary embodiments in which there are two participants to an instant messaging session, the instant invention may be readily employed in instant messaging sessions involving three or more participants. In one embodiment, all locally generated messages are presented on the right side 406 of the display window in the GUI, and all remotely generated messages are presented on the left side 404 of the display window. Thus, where there are two or more remote participants, each of their messages are presented on the left side 404 of the local participant's display window. In other embodiments, each remote participant's messages could be displayed in a spatially distinct region from that of the other participants. For example, messages from first, second and third remote participants could be displayed in first, second and third regions, respectively, wherein the first, second and third regions are spatially distinct. In the embodiment shown in FIG. 5B, a third participant is assigned to a central region, with his/her_icon appearing in the central region and the associated speech bubble extending generally therefrom.


Additionally, while the embodiments described herein have been shown with the GUI divided into spatially distinct horizontal regions, it is envisioned that other embodiments may spatially distinguish the various participants in other manners without departing from the spirit and scope of the instant invention. For example, the various participants may be distinguished by dividing the GUI into spatially distinct vertical regions with each participant being assigned a vertical portion of the GUI. For example, the local participant may be assigned the top region of the GUI, and the remote participant may be assigned the bottom region. Additional remote participants may be grouped together or, as described above, assigned an intermediate region, such as a vertical middle region. In a GUI with vertically distinct regions it may also be useful to allow the conversation to scroll horizontally, as opposed to the common vertical scrolling employed in many instant messaging systems. For example, more recent messages may be displayed to the right of older messages, with older messages scrolling off the left side of the GUI as the conversation advances.


If the messaging session is complete, such as by one or both of the parties logging off of the network or otherwise shutting down the software, then block 318 detects the ending of the session and transfers control out to another program responsible for a proper and orderly winding up of the program. Otherwise, if the instant messaging session continues, then control transfers back to block 302 where the process repeats.


Turning now to FIG. 6, an alternative embodiment of at least a portion of the software shown in FIG. 3 is illustrated. In this embodiment of the software, provision is made to preserve the order of a conversation during those instances in which a first party is preparing a response but the second party nevertheless sends a response before the first party completes and sends its response. For example, consider the instant messaging session shown in FIG. 7 to illustrate an out-of-order presentation of messages in a conversation. In the embodiment illustrated in FIG. 7, Sue 402 is in the process of preparing a response to a message 700 generated by Mary 400. Accordingly, a thought bubble 702 is positioned adjacent the Sue icon 402 below the message 700. Mary 400, however, did not wait for Sue's response, but sent a message 704. Thus, once Sue 402 completes and sends the response, the thought bubble 702 will be replaced by a speech balloon (not shown) containing the message. If the speech balloon (not shown) merely replaces the thought bubble with re-ordering, then the conversation will appear to have occurred in the order 700-702-704, even though the speech bubble replacing the thought bubble 702 occurred after, not before the speech bubble 704. This out-of-sequence ordering may give rise to confusion, particularly where the participants read the flow of the conversation at a subsequent time.


The flowchart of FIG. 6 illustrates one embodiment of a method useful in reordering the speech balloons so that they appear in the GUI in the order in which they actually occurred. Generally, the process set forth in FIG. 3 is substantially similar to that of FIG. 6, with the exception of blocks 600, 602, and 604. Generally, the order of the speech balloons is maintained based upon the time that the message was completed. Thought bubbles, on the other hand, are ordered based upon the time that they were created and are subsequently replaced by a speech balloon. Because a thought bubble may be created well before the corresponding speech balloon is completed, it is possible for other parties to complete messages in the intervening time. Thus, when the corresponding speech bubble is completed and replaces the corresponding thought bubble, the order of the speech balloons may vary.


At block 306, the software checks for a signal from the other remote party indicating that a message is being prepared. If such a signal is received, control transfers to block 600 where the GUI is activated to produce a graphical indication that a message is being prepared by the other party. The order in which the graphical indication is displayed is based upon the time that the thought bubble was created. The time that the thought bubble was created may be determined from a time stamp provided by the remote user who is in the process of preparing the message.


Thereafter, at block 310, the software checks to determine if a message has been received from the other party. If so, control transfers to block 602 where the software displays the text message (or emoticon, or the like) along with the icon associated with the author. In this instance, any corresponding thought bubble is removed and replaced by the corresponding speech balloon and its accompanying text. However, the speech balloon is ordered based upon the time completed. The time that the speech bubble was completed may be determined from a time stamp provided by the remote user who generated the message.


An exemplary instant messaging session that illustrates the results of this ordering policy is presented in FIGS. 7 and 8. In the embodiment illustrated in FIG. 7, Sue 402 is in the process of preparing a response to a message 700 generated by Mary 400. Accordingly, a thought bubble 702 is positioned adjacent the Sue icon 402 below the message 700. Mary 400, however, did not wait for Sue's response, but sent a message 704. Because the speech balloons 700, 704 are ordered based on the time completed whereas the thought bubble 702 is ordered based on the time created, the order of the messages will remain as shown in FIG. 7, until the message from Sue 402 is finally received.


Thus, as is shown in FIG. 8, a speech balloon 800 has replaced the thought bubble 702, but is located after (or below) the speech balloon 704, as the speech balloon 800 was completed after the speech balloon 704. In this manner, the actual order of the conversation is preserved.


A substantially similar process occurs with respect to displaying speech balloons associated with the local user. For example, at block 314, the software checks to determine if the message being prepared by the local party is complete. If so, control transfers to block 604 and the software delivers the message over the network connection to the other party. The message is then displayed in a speech balloon in an order based on the time that the message was completed.


Turning now to FIGS. 9 and 10, one exemplary method for formatting and sizing the speech balloon 410 is shown. In one embodiment, a text system, such as a standard text system used in Mac OS X is used to produce the text of the message. The text system provides information regarding the size of the text message to the GUI. The GUI uses this size information to construct a speech balloon of an appropriate size to contain the message. Certain rules regarding the formatting of the speech balloon affect the size of the speech balloon. For example, in one embodiment, upper, lower, left and right margins are established. In the exemplary embodiment illustrated in FIG. 9A, the margins are selected as follows: upper—3; lower—5; left—13; and right—6. The text of the message is required to be positioned within the speech balloon and within these margins. Using these margins causes the speech balloon to fit tightly around the message, as illustrated in FIG. 9B, so as to be aesthetically pleasing while not unnecessarily consuming large portions of the GUI. Reducing the size of the speech balloons allows more messages to appear on the GUI at one time.


The GUI uses the information regarding the size of the text message and the desired margins to produce a speech balloon of the appropriate size. The process involves dividing a template speech balloon into nine regions, such as is shown in FIG. 10A. The nine regions are comprised of four corners, left and right edges, top and bottom edges, and a central region. Initially, the margins are added to the rectangular area taken up by the text to produce a destination rectangle having dimensions in which the balloon is to be drawn. The four corner regions are drawn directly into the corners of the destination rectangle without significant change to their shape or size. The top and bottom edges are tiled horizontally into rectangles of the appropriate width (and original height). Varying the width of the top and bottom edges has the desirable effect of altering the horizontal dimension of the speech balloon. The left and right edges are tiled vertically into rectangles of the appropriate height (and original width) to produce a stretched or modified speech bubble 1000, as shown in FIG. 10B. It should be noted that the destination rectangle can be smaller than the original template image in either or both its vertical and horizontal dimension. For example, in the embodiment illustrated in FIG. 10B, the vertical dimension of the speech balloon is smaller than the vertical dimension of the template speech balloon of FIG. 10A, and the horizontal dimension of the speech balloon is larger than the horizontal dimension of the template speech balloon of FIG. 10A.


Once the speech balloon is appropriately sized, it is color filled according to the requirements of the sender or the recipient, as discussed above. Coloration and shading of the speech balloon is accomplished by alternative methodologies. In one embodiment, custom artwork is provided for each color to produce a desired variation across the surface of the speech balloon. For example, the color may be varied so that the coloration is lighter adjacent the bottom edge of the speech balloon. This coloration scheme has proven to be pleasing to users, providing the appearance of three dimensional qualities to the speech balloon. Alternatively, rather than developing custom artwork for each possible color, the speech balloon may be filled uniformly with the desired color. Thereafter, standard template shadows may be overlayed to produce a similar three dimensional effect.


Finally, while FIGS. 9 and 10 illustrate the speech balloon 410 drawn with its tail extending leftward, so as to be used in the left horizontal region of the GUI, the same processes described herein may be applied in drawing the speech balloon 410 with its tail extending rightward, so as to be used in the right horizontal region of the GUI. In particular, when the speech balloon 410 with a rightward extending tail is required, the speech balloon 410 with the leftward extending tail is designed to be of the proper size for the message, and then the speech balloon 410 is flipped horizontally or rotated about a central vertical axis to produce the appropriately sized speech balloon 410 with a rightward extending tail. In this manner, an economy of software coding is realized, as only a single routine for generating speech balloon with either leftward or rightward extending tails is required.


The particular embodiments disclosed above are illustrative only, as the invention may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. Furthermore, no limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope and spirit of the invention. Accordingly, the protection sought herein is as set forth in the claims below.

Claims
  • 1. An electronic device, comprising: a display;one or more processors; andmemory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, on the display, a communication session window, the communication session window being configured to display a plurality of instant messages during an instant messaging session between a local participant and a first remote participant, the communication session window having a first side and a second side different from the first side, the first side being associated with the first remote participant and the second side being associated with the local participant;receiving a signal indicating that the first remote participant is preparing an instant message;in response to receiving the signal indicating that the first remote participant is preparing an instant message, displaying, at a first location on the first side, a graphical indication that the first remote participant is preparing an instant message;receiving an instant message from the first remote participant;in response to receiving the instant message from the first remote participant: ceasing display of the graphical indication that the first remote participant is preparing an instant message; anddisplaying, at the first location on the first side at which the graphical indication that the first remote participant is preparing an instant message was previously displayed, a representation of the instant message from the first remote participant, wherein the representation of the instant message is different from the graphical indication that the first remote participant is preparing an instant message;receiving an instant message from the local participant; andin response to receiving the instant message from the local participant: displaying, at a second location on the second side, a representation of the instant message from the local participant.
  • 2. The electronic device of claim 1, wherein: the communication session window includes a text input field, andthe graphical indication that the first remote participant is preparing an instant message is displayed above the text input field.
  • 3. The electronic device of claim 1, wherein the graphical indication that the remote participant is preparing an instant message is displayed adjacent to a representation of the remote participant.
  • 4. The electronic device of claim 1, the one or more programs further including instructions for: receiving a signal indicating that a second remote participant of the instant messaging session is preparing an instant message; andin response to receiving the signal indicating that the second remote participant is preparing an instant message, displaying, at a second location on the display that is different from the first location, a graphical indication that the second remote participant is preparing an instant message.
  • 5. The electronic device of claim 1, the one or more programs further including instructions for: while displaying the graphical indication that the first remote participant is preparing an instant message, receiving an instant message from the local participant; andin response to receiving the instant message from the local participant, displaying, on the display, a representation of the instant message from the local participant.
  • 6. The electronic device of claim 1, wherein: the communication session window includes a representation of an instant message from the local participant, andthe graphical indication that the first remote participant is preparing an instant message is displayed below the representation of the instant message from the local participant.
  • 7. The electronic device of claim 1, wherein: the communication session window includes a representation of an instant message from the local participant, andthe representation of the instant message from the local participant is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 8. The electronic device of claim 1, wherein the representation of the instant message from the first remote participant is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 9. The electronic device of claim 1, wherein the graphical indication that the first remote participant is preparing an instant message is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 10. The electronic device of claim 1, wherein the first location on the display is on a side of the communication session window that corresponds to the first remote participant.
  • 11. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs including instructions for: displaying, on the display, a communication session window, the communication session window being configured to display a plurality of instant messages during an instant messaging session between a local participant and a first remote participant, the communication session window having a first side and a second side different from the first side, the first side being associated with the first remote participant and the second side being associated with the local participant;receiving a signal indicating that the first remote participant is preparing an instant message;in response to receiving the signal indicating that the first remote participant is preparing an instant message, displaying, at a first location on the first side, a graphical indication that the first remote participant is preparing an instant message;receiving an instant message from the first remote participant;in response to receiving the instant message from the first remote participant: ceasing display of the graphical indication that the first remote participant is preparing an instant message; anddisplaying, at the first location on the first side at which the graphical indication that the first remote participant is preparing an instant message was previously displayed, a representation of the instant message from the first remote participant, wherein the representation of the instant message is different from the graphical indication that the first remote participant is preparing an instant message;receiving an instant message from the local participant; andin response to receiving the instant message from the local participant; displaying, at a second location on the second side, a representation of the instant message from the local participant.
  • 12. The non-transitory computer-readable storage medium of claim 11, wherein: the communication session window includes a text input field, andthe graphical indication that the first remote participant is preparing an instant message is displayed above the text input field.
  • 13. The non-transitory computer-readable storage medium of claim 11, wherein the graphical indication that the remote participant is preparing an instant message is displayed adjacent to a representation of the remote participant.
  • 14. The non-transitory computer-readable storage medium of claim 11, the one or more programs further including instructions for: receiving a signal indicating that a second remote participant of the instant messaging session is preparing an instant message; andin response to receiving the signal indicating that the second remote participant is preparing an instant message, displaying, at a second location on the display that is different from the first location, a graphical indication that the second remote participant is preparing an instant message.
  • 15. The non-transitory computer-readable storage medium of claim 11, the one or more programs further including instructions for: while displaying the graphical indication that the first remote participant is preparing an instant message, receiving an instant message from the local participant; andin response to receiving the instant message from the local participant, displaying, on the display, a representation of the instant message from the local participant.
  • 16. The non-transitory computer-readable storage medium of claim 11, wherein: the communication session window includes a representation of an instant message from the local participant, andthe graphical indication that the first remote participant is preparing an instant message is displayed below the representation of the instant message from the local participant.
  • 17. The non-transitory computer-readable storage medium of claim 11, wherein: the communication session window includes a representation of an instant message from the local participant, andthe representation of the instant message from the local participant is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 18. The non-transitory computer-readable storage medium of claim 11, wherein the representation of the instant message from the first remote participant is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 19. The non-transitory computer-readable storage medium of claim 11, wherein the graphical indication that the first remote participant is preparing an instant message is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 20. The non-transitory computer-readable storage medium of claim 11, wherein the first location on the display is on a side of the communication session window that corresponds to the first remote participant.
  • 21. A method, comprising: at an electronic device with a display: displaying, on the display, a communication session window, the communication session window being configured to display a plurality of instant messages during an instant messaging session between a local participant and a first remote participant, the communication session window having a first side and a second side different from the first side, the first side being associated with the first remote participant and the second side being associated with the local participant;receiving a signal indicating that the first remote participant is preparing an instant message;in response to receiving the signal indicating that the first remote participant is preparing an instant message, displaying, at a first location on the first side, a graphical indication that the first remote participant is preparing an instant message;receiving an instant message from the first remote participant;in response to receiving the instant message from the first remote participant: ceasing display of the graphical indication that the first remote participant is preparing an instant message; anddisplaying, at the first location on the first side at which the graphical indication that the first remote participant is preparing an instant message was previously displayed, a representation of the instant message from the first remote participant, wherein the representation of the instant message is different from the graphical indication that the first remote participant is preparing an instant message;receiving an instant message from the local participant; andin response to receiving the instant message from the local participant: displaying, at a second location on the second side, a representation of the instant message from the local participant.
  • 22. The method of claim 21, wherein: the communication session window includes a text input field, andthe graphical indication that the first remote participant is preparing an instant message is displayed above the text input field.
  • 23. The method of claim 21, wherein the graphical indication that the remote participant is preparing an instant message is displayed adjacent to a representation of the remote participant.
  • 24. The method of claim 21, further comprising: receiving a signal indicating that a second remote participant of the instant messaging session is preparing an instant message; andin response to receiving the signal indicating that the second remote participant is preparing an instant message, displaying, at a second location on the display that is different from the first location, a graphical indication that the second remote participant is preparing an instant message.
  • 25. The method of claim 21, further comprising: while displaying the graphical indication that the first remote participant is preparing an instant message, receiving an instant message from the local participant; andin response to receiving the instant message from the local participant, displaying, on the display, a representation of the instant message from the local participant.
  • 26. The method of claim 21, wherein: the communication session window includes a representation of an instant message from the local participant, andthe graphical indication that the first remote participant is preparing an instant message is displayed below the representation of the instant message from the local participant.
  • 27. The method of claim 21, wherein: the communication session window includes a representation of an instant message from the local participant, andthe representation of the instant message from the local participant is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 28. The method of claim 21, wherein the representation of the instant message from the first remote participant is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 29. The method of claim 21, wherein the graphical indication that the first remote participant is preparing an instant message is displayed in a graphical element that is overlaid on the background of the communication session window.
  • 30. The method of claim 21, wherein the first location on the display is on a side of the communication session window that corresponds to the first remote participant.
CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No. 15/714,783, titled “METHOD AND APPARATUS FOR DISPLAYING INFORMATION DURING AN INSTANT MESSAGING SESSION,” filed Sep. 25, 2017, which is a continuation of U.S. patent application Ser. No. 13/908,998, titled “METHOD AND APPARATUS FOR DISPLAYING INFORMATION DURING AN INSTANT MESSAGING SESSION,” filed Jun. 3, 2013, which is a continuation of U.S. patent application Ser. No. 11/688,664, titled “METHOD AND APPARATUS FOR DISPLAYING INFORMATION DURING AN INSTANT MESSAGING SESSION,” filed Mar. 20, 2007, which is a divisional of U.S. patent application Ser. No. 10/428,523, titled “METHOD AND APPARATUS FOR DISPLAYING INFORMATION DURING AN INSTANT MESSAGING SESSION,” filed May 2, 2003. All of these applications are incorporated by reference herein in their entirety.

US Referenced Citations (413)
Number Name Date Kind
4107784 Van Bemmelen Aug 1978 A
4433377 Eustis et al. Feb 1984 A
4485439 Rothstein Nov 1984 A
4513379 Wilson et al. Apr 1985 A
4555775 Pike Nov 1985 A
4586158 Brandle Apr 1986 A
4642790 Minshull et al. Feb 1987 A
4974191 Amirghodsi et al. Nov 1990 A
5128672 Kaehler Jul 1992 A
5276794 Lamb, Jr. Jan 1994 A
5282265 Rohra Suda et al. Jan 1994 A
5287448 Nicol et al. Feb 1994 A
5386556 Hedin et al. Jan 1995 A
5428731 Powers, III Jun 1995 A
5434777 Luciw Jul 1995 A
5479488 Lennig et al. Dec 1995 A
5577241 Spencer Nov 1996 A
5608624 Luciw Mar 1997 A
5617539 Ludwig et al. Apr 1997 A
5682539 Conrad et al. Oct 1997 A
5724985 Snell et al. Mar 1998 A
5727950 Coo et al. Mar 1998 A
5748974 Johnson May 1998 A
5758079 Ludwig et al. May 1998 A
5794050 Dahlgren et al. Aug 1998 A
5815142 Allard et al. Sep 1998 A
5826261 Spencer Oct 1998 A
5854893 Ludwig et al. Dec 1998 A
5880731 Liles et al. Mar 1999 A
5884039 Ludwig et al. Mar 1999 A
5895466 Goldberg et al. Apr 1999 A
5896500 Ludwig et al. Apr 1999 A
5899972 Miyazawa et al. May 1999 A
5915249 Spencer Jun 1999 A
5917487 Ulrich Jun 1999 A
5943049 Matsubara et al. Aug 1999 A
5987404 Della Pietra et al. Nov 1999 A
5990887 Redpath Nov 1999 A
6018711 French-St. et al. Jan 2000 A
6052656 Suda et al. Apr 2000 A
6081750 Hoffberg et al. Jun 2000 A
6088731 Kiraly et al. Jul 2000 A
6144938 Surace et al. Nov 2000 A
6177931 Alexander et al. Jan 2001 B1
6188999 Moody Feb 2001 B1
6232966 Kurlander May 2001 B1
6233559 Balakrishnan, Sr. May 2001 B1
6237025 Ludwig et al. May 2001 B1
6246981 Papineni et al. Jun 2001 B1
6256663 Davis Jul 2001 B1
6268859 Andresen et al. Jul 2001 B1
6317594 Gossman et al. Nov 2001 B1
6317831 King Nov 2001 B1
6321092 Fitch et al. Nov 2001 B1
6334103 Surace et al. Dec 2001 B1
6351762 Ludwig et al. Feb 2002 B1
6421672 McAllister et al. Jul 2002 B1
6434524 Weber Aug 2002 B1
6434604 Harada et al. Aug 2002 B1
6437818 Ludwig et al. Aug 2002 B1
6446076 Burkey et al. Sep 2002 B1
6453292 Ramaswamy et al. Sep 2002 B2
6466654 Cooper et al. Oct 2002 B1
6499013 Weber Dec 2002 B1
6501937 Ho et al. Dec 2002 B1
6513063 Julia et al. Jan 2003 B1
6523061 Halverson et al. Feb 2003 B1
6526395 Morris Feb 2003 B1
6532444 Weber Mar 2003 B1
6532446 King Mar 2003 B1
6583806 Ludwig et al. Jun 2003 B2
6584494 Manabe Jun 2003 B1
6594673 Smith et al. Jul 2003 B1
6594688 Ludwig et al. Jul 2003 B2
6598039 Livowsky Jul 2003 B1
6601026 Appelt et al. Jul 2003 B2
6615172 Bennett et al. Sep 2003 B1
6630944 Kakuta et al. Oct 2003 B1
6631186 Adams et al. Oct 2003 B1
6633846 Bennett et al. Oct 2003 B1
6647260 Dusse et al. Nov 2003 B2
6650735 Burton et al. Nov 2003 B2
6662023 Helle Dec 2003 B1
6665639 Mozer et al. Dec 2003 B2
6665640 Bennett et al. Dec 2003 B1
6691111 Lazaridis et al. Feb 2004 B2
6691151 Cheyer et al. Feb 2004 B1
6704015 Bovarnick et al. Mar 2004 B1
6724370 Dutta et al. Apr 2004 B2
6735632 Kiraly et al. May 2004 B1
6742021 Halverson et al. May 2004 B1
6757362 Cooper et al. Jun 2004 B1
6757718 Halverson et al. Jun 2004 B1
6772195 Hatlelid et al. Aug 2004 B1
6775776 Vogt et al. Aug 2004 B1
6778951 Contractor Aug 2004 B1
6784901 Harvey et al. Aug 2004 B1
6792082 Levine Sep 2004 B1
6807574 Partovi et al. Oct 2004 B1
6810379 Vermeulen et al. Oct 2004 B1
6813491 McKinney Nov 2004 B1
6832194 Mozer et al. Dec 2004 B1
6842767 Partovi et al. Jan 2005 B1
6851115 Cheyer et al. Feb 2005 B1
6859931 Cheyer et al. Feb 2005 B1
6895380 Sepe, Jr. May 2005 B2
6895558 Loveland May 2005 B1
6910186 Kim Jun 2005 B2
6915138 Kraft Jul 2005 B2
6928614 Everhart Aug 2005 B1
6932708 Yamashita et al. Aug 2005 B2
6937975 Elworthy Aug 2005 B1
6954902 Noma et al. Oct 2005 B2
6964023 Maes et al. Nov 2005 B2
6980949 Ford Dec 2005 B2
6996531 Korall et al. Feb 2006 B2
6999927 Mozer et al. Feb 2006 B2
7013429 Fujimoto et al. Mar 2006 B2
7020685 Chen et al. Mar 2006 B1
7027974 Busch et al. Apr 2006 B1
7036128 Julia et al. Apr 2006 B1
7050977 Bennett May 2006 B1
7062428 Hogenhout et al. Jun 2006 B2
7069560 Cheyer et al. Jun 2006 B1
7072941 Griffin et al. Jul 2006 B2
7092887 Mozer et al. Aug 2006 B2
7092928 Elad et al. Aug 2006 B1
7115035 Tanaka Oct 2006 B2
7124164 Chemtob Oct 2006 B1
7127046 Smith et al. Oct 2006 B1
7133900 Szeto Nov 2006 B1
7136710 Hoffberg et al. Nov 2006 B1
7137126 Coffman et al. Nov 2006 B1
7139714 Bennett et al. Nov 2006 B2
7152093 Ludwig et al. Dec 2006 B2
7177798 Hsu et al. Feb 2007 B2
7197460 Gupta et al. Mar 2007 B1
7200559 Wang Apr 2007 B2
7203646 Bennett Apr 2007 B2
7206809 Ludwig et al. Apr 2007 B2
7216073 Lavi et al. May 2007 B2
7216080 Tsiao et al. May 2007 B2
7218943 Klassen et al. May 2007 B2
7225125 Bennett et al. May 2007 B2
7233790 Kjellberg et al. Jun 2007 B2
7233904 Luisi Jun 2007 B2
7266496 Wang et al. Sep 2007 B2
7277854 Bennett et al. Oct 2007 B2
7290039 Lisitsa et al. Oct 2007 B1
7299033 Kjellberg et al. Nov 2007 B2
7310600 Garner et al. Dec 2007 B1
7324947 Jordan et al. Jan 2008 B2
7349953 Lisitsa et al. Mar 2008 B2
7376556 Bennett May 2008 B2
7376645 Bernard May 2008 B2
7379874 Schmid et al. May 2008 B2
7386449 Sun et al. Jun 2008 B2
7386799 Clanton Jun 2008 B1
7392185 Bennett Jun 2008 B2
7398209 Kennewick et al. Jul 2008 B2
7403938 Harrison et al. Jul 2008 B2
7409337 Potter et al. Aug 2008 B1
7412470 Masuno et al. Aug 2008 B2
7415100 Cooper et al. Aug 2008 B2
7418392 Mozer et al. Aug 2008 B1
7423661 Abe et al. Sep 2008 B2
7426467 Nashida et al. Sep 2008 B2
7433921 Ludwig et al. Oct 2008 B2
7447635 Konopka et al. Nov 2008 B1
7454351 Jeschke et al. Nov 2008 B2
7467087 Gillick et al. Dec 2008 B1
7475010 Chao Jan 2009 B2
7478129 Chemtob Jan 2009 B1
7483894 Cao Jan 2009 B2
7487089 Mozer Feb 2009 B2
7502738 Kennewick et al. Mar 2009 B2
7519667 Capps Apr 2009 B1
7522927 Fitch et al. Apr 2009 B2
7523108 Cao Apr 2009 B2
7526466 Au Apr 2009 B2
7539656 Fratkina et al. May 2009 B2
7546382 Healey et al. Jun 2009 B2
7548895 Pulsipher Jun 2009 B2
7555431 Bennett Jun 2009 B2
7571106 Cao et al. Aug 2009 B2
7599918 Shen et al. Oct 2009 B2
7620407 Donald et al. Nov 2009 B1
7620549 Di Cristo et al. Nov 2009 B2
7624007 Bennett Nov 2009 B2
7634409 Kennewick et al. Dec 2009 B2
7636578 Cope et al. Dec 2009 B1
7640160 Di Cristo et al. Dec 2009 B2
7647225 Bennett et al. Jan 2010 B2
7657424 Bennett Feb 2010 B2
7669134 Christie et al. Feb 2010 B1
7672841 Bennett Mar 2010 B2
7676026 Baxter, Jr. Mar 2010 B1
7684985 Dominach et al. Mar 2010 B2
7685530 Sherrard et al. Mar 2010 B2
7693720 Kennewick et al. Apr 2010 B2
7698131 Bennett Apr 2010 B2
7702500 Blaedow Apr 2010 B2
7702508 Bennett Apr 2010 B2
7707027 Balchandran et al. Apr 2010 B2
7707032 Wang et al. Apr 2010 B2
7707267 Lisitsa et al. Apr 2010 B2
7710262 Ruha May 2010 B2
7711672 Au May 2010 B2
7716056 Weng et al. May 2010 B2
7720674 Kaiser et al. May 2010 B2
7720683 Vermeulen et al. May 2010 B1
7725307 Bennett May 2010 B2
7725318 Gavalda et al. May 2010 B2
7725320 Bennett May 2010 B2
7725321 Bennett May 2010 B2
7729904 Bennett Jun 2010 B2
7729916 Coffman et al. Jun 2010 B2
7734461 Kwak et al. Jun 2010 B2
7743188 Haitani et al. Jun 2010 B2
7752152 Paek et al. Jul 2010 B2
7774204 Mozer et al. Aug 2010 B2
7783486 Rosser et al. Aug 2010 B2
7801729 Mozer Sep 2010 B2
7809570 Kennewick et al. Oct 2010 B2
7809610 Cao Oct 2010 B2
7818176 Freeman et al. Oct 2010 B2
7822608 Cross et al. Oct 2010 B2
7831426 Bennett Nov 2010 B2
7840400 Lavi et al. Nov 2010 B2
7840447 Kleinrock et al. Nov 2010 B2
7873519 Bennett Jan 2011 B2
7873654 Bernard Jan 2011 B2
7877705 Chambers et al. Jan 2011 B2
7881936 Longe et al. Feb 2011 B2
7912702 Bennett Mar 2011 B2
7917367 Di Cristo et al. Mar 2011 B2
7917497 Harrison et al. Mar 2011 B2
7920678 Cooper et al. Apr 2011 B2
7930168 Weng et al. Apr 2011 B2
7949529 Weider et al. May 2011 B2
7974844 Sumita Jul 2011 B2
7974972 Cao Jul 2011 B2
7983915 Knight et al. Jul 2011 B2
7983917 Kennewick et al. Jul 2011 B2
7983997 Allen et al. Jul 2011 B2
7987151 Schott et al. Jul 2011 B2
8000453 Cooper et al. Aug 2011 B2
8005679 Jordan et al. Aug 2011 B2
8015006 Kennewick et al. Sep 2011 B2
8024195 Mozer et al. Sep 2011 B2
8036901 Mozer Oct 2011 B2
8041570 Mirkovic et al. Oct 2011 B2
8041611 Kleinrock et al. Oct 2011 B2
8055708 Chitsaz et al. Nov 2011 B2
8069046 Kennewick et al. Nov 2011 B2
8073681 Baldwin et al. Dec 2011 B2
8082153 Coffman et al. Dec 2011 B2
8095364 Longe et al. Jan 2012 B2
8099289 Mozer et al. Jan 2012 B2
8107401 John et al. Jan 2012 B2
8112275 Kennewick et al. Feb 2012 B2
8112280 Lu Feb 2012 B2
8140335 Kennewick et al. Mar 2012 B2
8150922 Griffin et al. Apr 2012 B2
8165886 Gagnon et al. Apr 2012 B1
8195467 Mozer et al. Jun 2012 B2
8204238 Mozer Jun 2012 B2
8219407 Roy et al. Jul 2012 B1
8458278 Christie et al. Jun 2013 B2
8554861 Christie et al. Oct 2013 B2
20010005859 Okuyama et al. Jun 2001 A1
20020007395 Matsuda Jan 2002 A1
20020035467 Morimoto et al. Mar 2002 A1
20020054094 Matsuda May 2002 A1
20030078969 Sprague et al. Apr 2003 A1
20030079024 Hough et al. Apr 2003 A1
20030097408 Kageyama et al. May 2003 A1
20030101450 Davidsson et al. May 2003 A1
20030122787 Zimmerman et al. Jul 2003 A1
20030130014 Rucinski Jul 2003 A1
20030134678 Tanaka Jul 2003 A1
20030152203 Berger et al. Aug 2003 A1
20030187925 Inala et al. Oct 2003 A1
20030208756 Macrae et al. Nov 2003 A1
20030228909 Tanaka et al. Dec 2003 A1
20040015548 Lee Jan 2004 A1
20040021676 Chen et al. Feb 2004 A1
20040039779 Amstrong et al. Feb 2004 A1
20040078441 Malik Apr 2004 A1
20040078445 Malik Apr 2004 A1
20040100479 Nakano et al. May 2004 A1
20040125088 Zimmerman et al. Jul 2004 A1
20040199663 Horvitz et al. Oct 2004 A1
20040210634 Ferrer et al. Oct 2004 A1
20040215731 Tzann-en Szeto Oct 2004 A1
20040223485 Arellano et al. Nov 2004 A1
20050071332 Ortega et al. Mar 2005 A1
20050080625 Bennett et al. Apr 2005 A1
20050086605 Ferrer et al. Apr 2005 A1
20050119897 Bennett et al. Jun 2005 A1
20050144568 Gruen et al. Jun 2005 A1
20050169213 Scian et al. Aug 2005 A1
20050181821 Elomaa Aug 2005 A1
20050210394 Crandall et al. Sep 2005 A1
20050278643 Ukai et al. Dec 2005 A1
20060001652 Chiu et al. Jan 2006 A1
20060026233 Tenembaum et al. Feb 2006 A1
20060122834 Bennett Jun 2006 A1
20060143007 Koh et al. Jun 2006 A1
20060152496 Knaven Jul 2006 A1
20060168539 Hawkins et al. Jul 2006 A1
20060174207 Deshpande Aug 2006 A1
20060184886 Chung et al. Aug 2006 A1
20060205432 Hawkins et al. Sep 2006 A1
20060253787 Fogg Nov 2006 A1
20070055529 Kanevsky et al. Mar 2007 A1
20070083623 Nishimura et al. Apr 2007 A1
20070088556 Andrew Apr 2007 A1
20070100790 Cheyer et al. May 2007 A1
20070132738 Lowles et al. Jun 2007 A1
20070162589 Riddle Jul 2007 A1
20070173267 Klassen et al. Jul 2007 A1
20070174188 Fish Jul 2007 A1
20070182595 Ghasabian Aug 2007 A1
20070185917 Prahlad et al. Aug 2007 A1
20070213099 Bast Sep 2007 A1
20070226652 Kikuchi et al. Sep 2007 A1
20070271527 Paas et al. Nov 2007 A1
20070282595 Tunning et al. Dec 2007 A1
20070300140 Makela et al. Dec 2007 A1
20080015864 Ross et al. Jan 2008 A1
20080021708 Bennett et al. Jan 2008 A1
20080034032 Healey et al. Feb 2008 A1
20080052063 Bennett et al. Feb 2008 A1
20080114841 Lambert May 2008 A1
20080120112 Jordan et al. May 2008 A1
20080140657 Azvine et al. Jun 2008 A1
20080221903 Kanevsky et al. Sep 2008 A1
20080222118 Scian et al. Sep 2008 A1
20080228496 Yu et al. Sep 2008 A1
20080247519 Abella et al. Oct 2008 A1
20080300878 Bennett Dec 2008 A1
20090006343 Platt et al. Jan 2009 A1
20090007001 Morin et al. Jan 2009 A1
20090030800 Grois Jan 2009 A1
20090058823 Kocienda Mar 2009 A1
20090076796 Daraselia Mar 2009 A1
20090100049 Cao Apr 2009 A1
20090106376 Tom et al. Apr 2009 A1
20090138828 Schultz et al. May 2009 A1
20090150156 Kennewick et al. Jun 2009 A1
20090157401 Bennett Jun 2009 A1
20090164441 Cheyer Jun 2009 A1
20090171664 Kennewick et al. Jul 2009 A1
20090299745 Kennewick et al. Dec 2009 A1
20090299849 Cao et al. Dec 2009 A1
20100005081 Bennett Jan 2010 A1
20100023320 Di cristo et al. Jan 2010 A1
20100036660 Bennett Feb 2010 A1
20100042400 Block et al. Feb 2010 A1
20100145700 Kennewick et al. Jun 2010 A1
20100164897 Morin et al. Jul 2010 A1
20100204986 Kennewick et al. Aug 2010 A1
20100217604 Baldwin et al. Aug 2010 A1
20100228540 Bennett Sep 2010 A1
20100235341 Bennett Sep 2010 A1
20100257160 Cao Oct 2010 A1
20100277579 Cho et al. Nov 2010 A1
20100280983 Cho et al. Nov 2010 A1
20100286985 Kennewick et al. Nov 2010 A1
20100299142 Freeman et al. Nov 2010 A1
20100312547 Van os et al. Dec 2010 A1
20100318576 Kim Dec 2010 A1
20100332235 David Dec 2010 A1
20100332348 Cao Dec 2010 A1
20110082688 Kim et al. Apr 2011 A1
20110112827 Kennewick et al. May 2011 A1
20110112921 Kennewick et al. May 2011 A1
20110119049 Ylonen May 2011 A1
20110125540 Jang et al. May 2011 A1
20110131036 Dicristo et al. Jun 2011 A1
20110131045 Cristo et al. Jun 2011 A1
20110144999 Jang et al. Jun 2011 A1
20110161076 Davis et al. Jun 2011 A1
20110175810 Markovic et al. Jul 2011 A1
20110184730 Lebeau et al. Jul 2011 A1
20110214083 Kang Sep 2011 A1
20110218855 Cao et al. Sep 2011 A1
20110231182 Weider et al. Sep 2011 A1
20110231188 Kennewick et al. Sep 2011 A1
20110264643 Cao Oct 2011 A1
20110279368 Klein et al. Nov 2011 A1
20110306426 Novak et al. Dec 2011 A1
20120002820 Leichter Jan 2012 A1
20120016678 Gruber et al. Jan 2012 A1
20120020490 Leichter Jan 2012 A1
20120022787 Lebeau et al. Jan 2012 A1
20120022857 Baldwin et al. Jan 2012 A1
20120022860 Lloyd et al. Jan 2012 A1
20120022868 Lebeau et al. Jan 2012 A1
20120022869 Lloyd et al. Jan 2012 A1
20120022870 Kristjansson et al. Jan 2012 A1
20120022874 Lloyd et al. Jan 2012 A1
20120022876 Lebeau et al. Jan 2012 A1
20120023088 Cheng et al. Jan 2012 A1
20120034904 Lebeau et al. Feb 2012 A1
20120035908 Lebeau et al. Feb 2012 A1
20120035924 Jitkoff et al. Feb 2012 A1
20120035931 Lebeau et al. Feb 2012 A1
20120035932 Jitkoff et al. Feb 2012 A1
20120042343 Laligand et al. Feb 2012 A1
20140026073 Christie et al. Jan 2014 A1
20180191646 Christie et al. Jul 2018 A1
Foreign Referenced Citations (29)
Number Date Country
464712 Jan 1992 EP
1245023 Oct 2002 EP
1347361 Sep 2003 EP
1517228 Mar 2005 EP
6-19965 Jan 1994 JP
9-259063 Oct 1997 JP
9-265457 Oct 1997 JP
2000-59422 Feb 2000 JP
2000-99225 Apr 2000 JP
2001-125896 May 2001 JP
2002-24212 Jan 2002 JP
2002-82893 Mar 2002 JP
2002-351789 Dec 2002 JP
2003-517158 May 2003 JP
2003-233568 Aug 2003 JP
2005-86624 Mar 2005 JP
2005-92441 Apr 2005 JP
2009-36999 Feb 2009 JP
10-0776800 Nov 2007 KR
10-0810500 Mar 2008 KR
10-2008-0109322 Dec 2008 KR
10-2009-0086805 Aug 2009 KR
10-0920267 Oct 2009 KR
10-2011-0113414 Oct 2011 KR
199833111 Jul 1998 WO
2003056789 Jul 2003 WO
2006020305 Feb 2006 WO
2006129967 Dec 2006 WO
2011088053 Jul 2011 WO
Non-Patent Literature Citations (232)
Entry
Adium, “Aboutadium—Adium X—Trac”, available at <http://web.archive.org/web/20070819113247/http://trac.adiumx.com/wiki/AboutAdium>, retrieved on Nov. 25, 2011, 2 pages.
Advisory Action received for U.S. Appl. No. 13/908,998, dated Nov. 10, 2015, 3 pages.
Ahuja et al., “A Comparison of Application Sharing Mechanisms in Real-time Desktop Conferencing Systems”, At&T Bell Laboratories, 1990, pp. 238-248.
Alfred App, “Alfred”, available at <http://www.alfredapp.com/>, retrieved on Feb. 8, 2012, 5 pages.
Ambite et al., “Design and Implementation of the CALO Query Manager”, American Association for Artificial Intelligence, 2006, 8 pages.
Ambite et al., “Integration of Heterogeneous Knowledge Sources in the CALO Query Manager”, The 4th International Conference on Ontologies, DataBases, and Applications of Semantics (ODBASE), 2005, 18 pages.
Apple Computer, Inc., “Welcome to Tiger”, available at <http://www.maths.dundee.ac.uk/software/Welcome_to_Mac_OS_X_v10.4_Tiger.pdf>, 2005, pp. 1-32.
Arango et al., “Touring Machine: a Software Platform for Distributed Multimedia Applications”, 1992 IFIP International Conference on Upper Layer Protocols, Architectures, and Applications, pp. 1-11.
Belvin et al., “Development of the HRL Route Navigation Dialogue System”, Proceedings of the First International Conference on Human Language Technology Research, Paper, 2001, 5 pages.
Berry et al., “PTIME: Personalized Assistance for Calendaring”, ACM Transactions on Intelligent Systems and Technology, vol. 2, No. 4, Article 40, Jul. 2011, pp. 1-22.
Betts et al., “Goals and Objectives for User Interface Software”, Computer Graphics, vol. 21, No. 2, Apr. 1987, pp. 73-78.
Bleher et al., “A Graphic Interactive Application Monitor”, IBM Systems Journal, vol. 19, No. 3, Sep. 1980, pp. 382-402.
Borenstein et al., “Cooperative Work in the Andrew Message System”, ACM Digital Library, 1988, pp. 306-323.
Bussler et al., “Web Service Execution Environment (WSMX)”, retrieved from Internet on Sep. 17, 2012, available at <http://www.w3.org/Submission/WSMX>, Jun. 3, 2005, 29 pages.
Butcher, Mike, “EVI Arrives in Town to Go Toe-to-toe with Siri”, TechCrunch, Jan. 23, 2012, pp. 1-2.
Buxton et al., “EuroPARC'S Integrated Interactive Intermedia Facility (IIIF): Early Experiences”, Proceedings of the IFIP WG 8.4 Conference on Multi-User Interfaces and Applications, 1990, pp. 11-34.
Casner et al., “N-way Conferencing with Packet Video”, The Third International Workshop on Packet Video, Mar. 22-23, 1990, pp. 1-6.
Chartier, David, “Using Multi-network Meebo Chat Service on Your iPhone”, available at <http://www.tuaw.com/2007/07/04/using-multi-network-meebo-chat-service-on-your-iphone/>, Jul. 4, 2007, 5 pages.
Chen, Yi, “Multimedia Siri Finds and Plays whatever You Ask for”, PSFK Report, Feb. 9, 2012, pp. 1-9.
Cheyer et al., “Spoken Language and Multimodal Applications for Electronic Realties”, Virtual Reality, vol. 3, 1999, pp. 1-15.
Cheyer, Adam, “A Perspective on Ai & Agent Technologies for SCM”, VerticalNet presentation, 2001, 22 pages.
Cheyer, Adam, “About Adam Cheyer”, available at: <http://www.adam.cheyer.com/about.html>, retrieved on Sep. 17, 2012, pp. 1-2.
Conklin, Jeffrey, “A Survey of Hypertext”, MCC Software Technology Program, Dec. 1987, 40 pages.
Croft et al., “Task Support in an Office System”, Proceedings of the Second ACM-SIGOA Conference on Office Information Systems, 1984, pp. 22-24.
Crowley et al., “MMCONF: an Infrastructure for Building Shared Multimedia Applications”, CSCW 90 Proceedings, Oct. 1990, pp. 329-342.
Cutkosky et al., “Pact: an Experiment in Integrating Concurrent Engineering Systems”, Journal, Computer, vol. 26, No. 1, Jan. 1993, pp. 0-13.
Decision to Grant received for Chinese Patent Application No. 200780041351.5, dated Jun. 12, 2013, 3 pages.
Decision to Grant received for Japanese Patent Application No. 2009-527503, dated Oct. 12, 2012, 3 pages.
Decision to Grant received for Japanese Patent Application No. 2012-246631, dated May 11, 2015, 6 pages.
Del Strother, Jonathan, “Coverflow”, Available online at <http://www.steelskies.com/coverflow>, retrieved on Jun. 15, 2006, 14 pages.
Domingue et al., “Web Service Modeling Ontology (WSMO)—an Ontology for Semantic Web Services”, Position Paper at the W3C Workshop on Frameworks for Semantics in Web Services, Innsbruck, Austria, Jun. 2005, 6 pages.
Donahue et al., “Whiteboards: a Graphical Database Tool”, ACM Transactions on Office Information Systems, vol. 4, No. 1, Jan. 1986, pp. 24-41.
Dourish et al., “Portholes: Supporting Awareness in a Distributed Work Group”, Proceedings of the SIGCHI conference on Human factors in Computing Systems, May 1992, pp. 541-547.
Egido, Carmen, “Video Conferencing as a Technology to Support Group Work: a Review of Its Failures”, Bell Communications Research, 1988, pp. 13-24.
Elio et al., “On Abstract Task Models and Conversation Policies”, Proc. Workshop on Specifying and Implementing Conversation Policies, Autonomous Agents'99 Conference, 1999, pp. 1-10.
Elliot, Chip, “High-quality Multimedia Conferencing Through a Long-haul Packet Network”, BBN Systems and Technologies, 1993, pp. 91-98.
Ericsson et al., “Software Illustrating a Unified Approach to Multimodality and Multilinguality in the In-home Domain”, Talk and Look: Tools for Ambient Linguistic Knowledge, Dec. 2006, 127 pages.
EVI, “Meet EVI: the One Mobile Application that Provides Solutions for Your Everyday Problems”, Feb. 2012, 3 pages.
Extended European Search Report (includes Partial European Search Report and European Search Opinion) received for European Patent Application No. 13155688.8, dated Aug. 22, 2013, 11 pages.
Feigenbaum et al., “Computer-assisted Semantic Annotation of Scientific Life Works”, Oct. 15, 2007, 22 pages.
Final Office Action received for U.S. Appl. No. 10/428,523, dated Apr. 22, 2009, 22 pages.
Final Office Action received for U.S. Appl. No. 10/428,523, dated Nov. 29, 2007, 19 pages.
Final Office Action received for U.S. Appl. No. 11/688,664, dated Feb. 22, 2010, 18 pages.
Final Office Action received for U.S. Appl. No. 11/688,664, dated Mar. 15, 2011, 19 pages.
Final Office Action received for U.S. Appl. No. 11/848,208, dated Nov. 4, 2011, 20 pages.
Final Office Action received for U.S. Appl. No. 11/848,208, dated Oct. 9, 2014, 15 pages.
Final Office Action received for U.S. Appl. No. 12/163,908, dated Nov. 7, 2012, 15 pages.
Final Office Action received for U.S. Appl. No. 12/242,846, dated Jun. 5, 2012, 14 pages.
Final Office Action received for U.S. Appl. No. 12/726,247, dated Mar. 18, 2011, 28 pages.
Final Office Action received for U.S. Appl. No. 13/908,998, dated Aug. 17, 2015, 23 pages.
Fisher et al., “Virtual Environment Display System”, Interactive 3D Graphics, Oct. 23-24, 1986, pp. 77-87.
Forsdick, Harry, “Explorations into Real-time Multimedia Conferencing”, Proceedings of the IFIP Tc 6 International Symposium on Computer Message Systems, 1986, pp. 331-347.
Gannes, Liz, “Alfred App Gives Personalized Restaurant Recommendations”, AllThingsD, Jul. 18, 2011, pp. 1-3.
Gardner, Jr., P. C., “A System for the Automated Office Environment”, IBM Systems Journal, vol. 20, No. 3, 1981, pp. 321-345.
Gautier et al., “Generating Explanations of Device Behavior Using Compositional Modeling and Causal Ordering”, CiteSeerx, 1993, pp. 89-97.
Gaver et al., “One is Not Enough: Multiple Views in a Media Space”, INTERCHI, Apr. 24-29, 1993, pp. 335-341.
Gaver et al., “Realizing a Video Environment: EUROPARC'S Rave System”, Rank Xerox Cambridge EuroPARC's, 1992, pp. 27-35.
Gervasio et al., “Active Preference Learning for Personalized Calendar Scheduling Assistancae”, CiteSeerx, Proceedings of IUI'05, Jan. 9-12, 2005, pp. 90-97.
Glass et al., “Multilingual Spoken-language Understanding in the MIT Voyager System”, Available online at <http://groups.csail.mitedu/sls/publications/1995/speechcomm95-voyager.pdf>, Aug. 1995, pp. 1-29.
Glass, Alyssa, “Explaining Preference Learning”, CiteSeerx, 2006, pp. 1-5.
Gmail, “About Group Chat”, available at <http://mail.google.com/support/bin/answer.py?answer=81090>, Nov. 26, 2007, 2 pages.
Goddeau et al., “A Form-based Dialogue Manager for Spoken Language Applications”, http://phasedance.com/pdf!icslp96.pdf, Oct. 1996, 4 pages.
Goddeau et al., “Galaxy: a Human-language Interface to On-line Travel Information”, International Conference on Spoken Language Processing, Yokohama, 1994, pp. 707-710.
Good et al., “Building a User-derived Interface”, Communications of the ACM; vol. 27, No. 10, Oct. 1984, pp. 1032-1043.
Gruber et al., “An Ontology for Engineering Mathematics”, Fourth International Conference on Principles of Knowledge Representation and Reasoning, Available at <http://www-ksl.stanford.edu/knowledge-sharing/papers/engmath.html>, 1994, pp. 1-22.
Gruber et al., “Generative Design Rationale: Beyond the Record and Replay Paradigm”, Knowledge Systems Laboratory, Technical Report KSL 92-59, Dec. 1991, Updated Feb. 1993, 24 pages.
Gruber et al., “Machine-generated Explanations of Engineering Models: a Compositional Modeling Approach”, Proceedings of International Joint Conference on Artificial Intelligence, 1993, 7 pages.
Gruber et al., “Nike: a National Infrastructure for Knowledge Exchange”, A Whitepaper Advocating and ATP Initiative on Technologies for Lifelong Learning, Oct. 1994, pp. 1-10.
Gruber et al., “Toward a Knowledge Medium for Collaborative Product Development”, Proceedings of the Second International Conference on Artificial Intelligence in Design, Jun. 22-25, 1992, pp. 1-19.
Gruber Thomas R. et al., “A Translation Approach to Portable Ontology Specifications”, Knowledge Acquisition, vol. 5, No. 2, Jun. 1993, pp. 199-220.
Gruber, Thomas R., “Automated Knowledge Acquisition for Strategic Knowledge”, Machine Learning, vol. 4, 1989, pp. 293-336.
Gruber, Thomas R., “Interactive Acquisition of Justifications: Learning “why” by Being Told “what””, Knowledge Systems Laboratory, Technical Report KSL 91-17, Original Oct. 1990, Revised Feb. 1991, 24 pages.
Gruber, Thomas R., “Toward Principles for the Design of Ontologies Used for Knowledge Sharing”, International Journal of Human-Computer Studies, vol. 43, No. 5-6, Nov. 1995, pp. 907-928.
Gruber, Tom, “(avoiding) the Travesty of the Commons”, Presentation at NPUC, New Paradigms for User Computing, IBM Almaden Research Center, Jul. 24, 2006, 52 pages.
Gruber, Tom, “2021: Mass Collaboration and the Really New Economy”, TNTY Futures, vol. 1, No. 6, Available at <http://tomgruber.org/writing/tnty2001.htm>, Aug. 2001, 5 pages.
Gruber, Tom, “Big Think Small Screen: How Semantic Computing in the Cloud Will Revolutionize the Consumer Experience on the Phone”, Keynote presentation at Web 3.0 conference, Jan. 2010, 41 pages.
Gruber, Tom, “Collaborating Around Shared Content on the Www, W3c Workshop on Www and Collaboration”, available at <http://www.w3.org/Collaboration/VVorkshop/Proceedings/P9.html>, Sep. 1995, 1 page.
Gruber, Tom, “Collective Knowledge Systems: Where the Social Web Meets the Semantic Web”, Web Semantics: Science, Services and Agents on the World Wide Web, 2007, pp. 1-19.
Gruber, Tom, “Despite Our Best Efforts, Ontologies Are Not the Problem”, AAAI Spring Symposium, Available at <http://tomgruber.org/writing/aaai-ss08.htm>, Mar. 2008, pp. 1-40.
Gruber, Tom, “Enterprise Collaboration Management with Intraspect”, Intraspect Technical White Paper, Jul. 2001, pp. 1-24.
Gruber, Tom, “Every Ontology is a Treaty-a Social Agreement-among People with Some Common Motive in Sharing”, Official Quarterly Bulletin of AIS Special Interest Group on Semantic Web and Information Systems, vol. 1, No. 3, 2004, pp. 1-5.
Gruber, Tom, “Helping Organizations Collaborate, Communicate, and Learn”, Presentation to NASA Ames Research, available at <http://tomgruber.org/writing/organizational-intelligence-talk.htm>, Mar.-Oct. 2003, 30 pages.
Gruber, Tom, “Intelligence at the Interface: Semantic Technology and the Consumer Internet Experience”, Presentation at Semantic Technologies Conference, Available online at <http://tomgruber.org/writing/semtech08.htm>, May 20, 2008, pp. 1-40.
Gruber, Tom, “It is What It Does: the Pragmatics of Ontology for Knowledge Sharing”, Proceedings of the International CIDOC CRM Symposium, Available at <http://tomgruber.org/writing/cidoc-ontology.htm>, Mar. 26, 2003, 21 pages.
Gruber, Tom, “Ontologies, Web 2.0 and Beyond”, Ontology Summit, available at <http://tomgruber.org/writing/ontolog-social-web-keynote.htm>, Apr. 24, 2007, 17 pages.
Gruber, Tom, “Ontology of Folksonomy: a Mash-up of Apples and Oranges”, Int'l Journal on Semantic Web & Information Systems, vol. 3, No. 2, 2007, 7 pages.
Gruber, Tom, “Siri a Virtual Personal Assistant-bringing Intelligence to the Interface”, Semantic Technologies conference, Jun. 16, 2009, 21 pages.
Gruber, Tom, “TagOntology”, Presentation to Tag Camp, Oct. 29, 2005, 20 pages.
Gruber, Tom, “Where the Social Web Meets the Semantic Web”, Presentation at the 5th International Semantic Web Conference, Nov. 2006, 38 pages.
Gruhn et al., “A Research Perspective on Computer-assisted Office Work”, IBM Systems Journal, vol. 18, No. 3, 1979, pp. 432-456.
Guzzoni et al., “A Unified Platform for Building Intelligent Web Interaction Assistants”, Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Computer Society, 2006, 4 pages.
Guzzoni et al., “Active, a Platform for Building Intelligent Operating Rooms”, Surgetica 2007 Computer-Aided Medical Interventions: tools and Applications, 2007, pp. 191-198.
Guzzoni et al., “Active, a Tool for Building Intelligent User Interfaces”, ASC 2007, Palma de Mallorca, Aug. 2007, 6 pages.
Guzzoni et al., “Modeling Human-agent Interaction with Active Ontologies”, AAAI Spring Symposium, Interaction Challenges for Intelligent Assistants, Stanford University, Palo Alto, California, 2007, 8 pages.
Halbert, D. C., “Programming by Example”, Dept. Electrical Engineering and Comp. Sciences, University of California, Berkley, Nov. 1984, 81 pages.
Hardwar, Devemder, “Driving App Waze Builds Its Own Siri for Hands-free Voice Control”, Available online at <http://venturebeat.com/2012/02/09/driving-app-waze-builds-its-own-siri-for-hands-free-voice-control/>, retrieved on Feb. 9, 2012, 4 pages.
Hill, R. D., “Some Important Features and Issues in User Interface Management System”, Dynamic Graphics Project, University of Toronto, CSRI, vol. 21, No. 2, Apr. 1987, pp. 116-120.
Hiroshi, “Teamwork Station: Towards a Seamless Shared Workspace”, NTT Human Interface Laboratories, CSCW 90 Proceedings, Oct. 1990, pp. 13-26.
Hopper, Andy, “Pandora—an Experimental System for Multimedia Applications”, Olivetti Research Laboratory, Apr. 1990, pp. 19-34.
Howard, John H., “(abstract) an Overview of the Andrew File System”, Information Technology Center, Carnegie Mellon University; (CMU-ITC-88-062) to Appear in a future issue of the ACM Transactions on Computer Systems, 1988, pp. 1-6.
Huang et al., “Real-time Software-based Video Coder for Multimedia Communication Systems”, Department of Computer Science and Information Engineering, 1993, pp. 1-10.
iChat AV, “Video Conferencing for the Rest of Us”, Apple—Mac OS X—iChat AV, available at <http://www.apple.com/macosx/features/ichat/html>, retrieved on Apr. 13, 2006, 3 pages.
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2007/077424, dated Mar. 10, 2009, 9 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2007/077424, dated Jun. 19, 2008, 13 pages.
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2011/020861, dated Nov. 29, 2011, 15 pages.
Intraspect Software, “The Intraspect Knowledge Management Solution: Technical Overview”, available at <http://tomgruber.org/writing/intraspect-whitepaper-1998.pdf>, 1998, 18 pages.
Invitation to Pay Additional Fees and Partial Search Report received for PCT Application No. PCT/US2007/077424, dated Apr. 29, 2008, 6 pages.
iPHONE Hacks, “Native iPhone Mms Application Released”, available at <http://www.iphonehacks.com/2007/12/iphone-mms-app.html>, retrieved on Dec. 25, 2007, 5 pages.
iPHONECHAT, “iChat for iPhone in Javascript”, available at <http://www.publictivity.com/iPhoneChat/>, retrieved on Dec. 25, 2007, 2 pages.
Jaybird, “Everything Wrong with Aim: because We've All Thought About It”, available at <http://www.psychonoble.com/archives/articles/82.html>, May 24, 2006, 3 pages.
Jeffay et al., “Kernel Support for Live Digital Audio and Video”, In Proc. of the Second Intl. Workshop on Network and Operating System Support for Digital Audio and Video, vol. 614, Nov. 1991, pp. 1-16.
Julia et al., “Un Editeur Interactif De Tableaux Dessines a Main Levee (an Interactive Editor for Hand-sketched Tables)”, Traitement du Signal, vol. 12, No. 6, 1995, pp. 619-626.
Kaeppner et al., “Architecture of Heiphone: a Testbed for Audio/video Teleconferencing”, IBM European Networking Center, 1993, 51 pages.
Karp, P. D., “A Generic Knowledge-base Access Protocol”, Available online at <http://lecture.cs.buu.ac.th/-f50353/Document/gfp.pdf>, May 12, 1994, 66 pages.
Kirstein et al., “Piloting of Multimedia Integrated Communications for European Researchers”, Proc. INET '93, 1993, pp. 1-12.
Kjelldahl et al., “Multimedia—Principles, Systems, and Applications”, Proceedings of the 1991 Eurographics Workshop on Multimedia Systems, Applications, and Interaction, Apr. 1991, 14 pages.
Knownav, “Knowledge Navigator”, YouTube Video available at <http://www.youtube.com/watch?v=QRH8eimU_20>, Apr. 29, 2008, 1 page.
Kurlander et al., “Comic Chat”, [Online], 1996 [Retrieved on: Feb. 4, 2013], SIGGRAPH '96 Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, [Retrieved from: http://delivery.acm.org/10.1145/240000/237260/p225-kurlander.pdf], 1996, pp. 225-236.
Lantz et al., “Towards a Universal Directory Service”, Departments of Computer Science and Electrical Engineering, Stanford University, 1985, pp. 250-260.
Lantz, Keith, “An Experiment in Integrated Multimedia Conferencing”, Department of Computer Science, Stanford University, 1986, pp. 267-275.
Lauwers et al., “Collaboration Awareness in Support of Collaboration Transparency: Requirements for the Next Generation of Shared Window Systems”, CHI'90 Proceedings, 1990, pp. 303-311.
Lauwers et al., “Replicated Architectures for Shared Window Systems: a Critique”, COCS '90 Proceedings of the ACM SIGOIS and IEEE CS TC-OA conference on Office information systems, ACM SIGOIS Bulletin, 1990, pp. 249-260.
Lemon et al., “Multithreaded Context for Robust Conversational Interfaces: Context-Sensitive Speech Recognition and Interpretation of Corrective Fragments”, ACM Transactions on Computer-Human Interaction, vol. 11, No. 3, Sep. 2004, pp. 241-267.
Leong et al., “CASIS: a Context-aware Speech Interface System”, Proceedings of the 10th International Conference on Intelligent user Interfaces, Jan. 9-12, 2005, pp. 231-238.
Lieberman et al., “Out of Context: Computer Systems that Adapt to, and Learn From, Context”, IBM Systems Journal, vol. 39, Nos. 3&4, 2000, pp. 617-632.
Lin et al., “A Distributed Architecture for Cooperative Spoken Dialogue Agents with Coherent Dialogue State and History”, Available on at: <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=1 0.1.1.42.272>, 1999, 4 pages.
McGuire et al., “Shade: Technology for Knowledge-based Collaborative Engineering”, Journal of Concurrent Engineering˜r Applications and Research (CERA), 1993, 18 pages.
Meng et al., “Wheels: a Conversational System in the Automobile Classified Domain”, Proceedings of Fourth International Conference on Spoken Language, ICSLP 96, vol. 1, Oct. 1996, pp. 1-4.
Meyer, Mike, “A Shell for Modern Personal Computers”, University of California, Aug. 1987, pp. 13-19.
Meyrowitz et al., “Bruwin: an Adaptable Design Strategy for Window Manager/virtual Terminal Systems”, Department of Computer Science, Brown University, 1981, pp. 180-189.
Milward et al., “D2.2: Dynamic Multimodallnterface Reconfiguration Alk and Look: Tools for Ambient Linguistic Knowledge”, available at <http://www.ihmc.us/users/nblaylock!Pubs/Files/talk d2.2.pdf>, Aug. 8, 2006, 69 pages.
Mitra et al., “A Graph-oriented Model for Articulation of Ontology Interdependencies”, Advances in Database Technology, Lecture Notes in Computer Science, vol. 1777, 2000, pp. 1-15.
Moran et al., “Multimodal User Interfaces in the Open Agent Architecture”, International Conference on Intelligent User Interfaces (IUI97), 1997, 8 pages.
Morland, D. Verne., “Human Factors Guidelines for Terminal Interface Design”, Communications of the ACM vol. 26, No. 7, Jul. 1983, pp. 484-494.
Morris et al., “Andrew: a Distributed Personal Computing Environment”, Communications of the ACM, vol. 29, No. 3, Mar. 1986, pp. 184-201.
Mozer, Michael C., “An Intelligent Environment must Be Adaptive”, IEEE Intelligent Systems, Mar./Apr. 1999, pp. 11-13.
Muller et al., “Cscw'92 Demonstrations”, 1992, pp. 11-14.
Naone, Erica, “Tr10: Intelligent Software Assistant”, Technology Review, Mar.-Apr. 2009, 2 pages.
Neches et al., “Enabling Technology for Knowledge Sharing”, Fall, 1991, pp. 37-56.
Non-Final Office Action received for U.S. Appl. No. 10/428,523, dated Apr. 12, 2007, 11 pages.
Non-Final Office Action received for U.S. Appl. No. 10/428,523, dated Jul. 9, 2008, 19 pages.
Non-Final Office Action received for U.S. Appl. No. 11/688,664, dated Jul. 19, 2012, 16 pages.
Non-Final Office Action received for U.S. Appl. No. 11/688,664, dated Jul. 22, 2009, 16 pages.
Non-Final Office Action received for U.S. Appl. No. 11/688,664, dated Sep. 29, 2010, 23 pages.
Non-Final Office Action received for U.S. Appl. No. 11/848,208, dated Dec. 23, 2013, 13 pages.
Non-Final Office Action received for U.S. Appl. No. 11/848,208, dated Apr. 1, 2011, 8 pages.
Non-Final Office Action received for U.S. Appl. No. 12/163,908, dated Jun. 13, 2012, 17 pages.
Non-Final Office Action received for U.S. Appl. No. 12/242,846, dated Oct. 19, 2011, 17 pages.
Non-Final Office Action received for U.S. Appl. No. 12/726,247, dated Jul. 18, 2012, 23 pages.
Non-Final Office Action received for U.S. Appl. No. 12/726,247, dated Sep. 28, 2010, 27 pages.
Non-Final Office Action received for U.S. Appl. No. 13/908,998, dated Mar. 3, 2015, 26 pages.
Non-Final Office Action received for U.S. Appl. No. 15/714,783, dated Jun. 27, 2018, 14 pages.
Notenboom, Leo A., “Can I Retrieve Old Msn Messenger Conversations?”, available at <http://ask-leo.com/can_i_retrieve_old_msn_messenger_conversations.html>, Mar. 11, 2004, 23 pages.
Noth et al., “Verbmobil: the Use of Prosody in the Linguistic Components of a Speech Understanding System”, IEEE Transactions on Speech and Audio Processing, vol. 8, No. 5, Sep. 2000, pp. 519-532.
Notice of Allowance received for Canadian Patent Application No. 2,661,886, dated Jan. 7, 2014, 1 page.
Notice of Allowance received for U.S. Appl. No. 10/428,523, dated Oct. 5, 2009, 10 pages.
Notice of Allowance received for U.S. Appl. No. 11/688,664, dated Feb. 8, 2013, 12 pages.
Notice of Allowance received for U.S. Appl. No. 12/242,846, dated Dec. 7, 2012, 9 pages.
Notice of Allowance received for U.S. Appl. No. 12/242,846, dated Feb. 5, 2013, 9 pages.
Notice of Allowance received for U.S. Appl. No. 12/726,247, dated Feb. 8, 2013, 10 pages.
Notice of Allowance received for U.S. Appl. No. 12/726,247, dated May 30, 2013, 10 pages.
Notice of Allowance received for U.S. Appl. No. 15/714,783, dated Dec. 4, 2018, 7 pages.
Notice of Allowance received for U.S. Appl. No. 15/714,783, dated Feb. 25, 2019, 16 pages.
Notice of Intent to Issue a Reexam Certificate received for U.S. Appl. No. 90/013,163, dated Jan. 29, 2015, 22 pages.
Notice of Restriction Requirement received for U.S. Appl. No. 10/428,523, dated Dec. 29, 2006, 4 pages.
Office Action received for Canadian Patent Application No. 2,661,886, dated Jul. 14, 2010, 3 pages.
Office Action received for Canadian Patent Application No. 2,661,886, dated Nov. 7, 2011, 3 pages.
Office Action received for Chinese Patent Application No. 200780041351.5, dated Aug. 3, 2012, 6 pages.
Office Action received for Chinese Patent Application No. 200780041351.5, dated Dec. 6, 2010, 5 pages.
Office Action received for Chinese Patent Application No. 200780041351.5, dated Nov. 3, 2011, 14 pages.
Office Action received for European Patent Application No. 07841749.0, dated Feb. 18, 2011, 4 pages.
Office Action received for European Patent Application No. 07841749.0, dated Nov. 14, 2012, 5 pages.
Office Action received for Japanese Patent Application No. 2009-527503, dated Sep. 16, 2011, 7 pages.
Office Action received for Japanese Patent Application No. 2012-246631, dated Nov. 18, 2013, 4 pages.
Office Action received for Japanese Patent Application No. 2012-246631, dated Oct. 17, 2014, 5 pages.
Padilla, Alfredo, “Palm Treo 750 Cell Phone Review—Messaging”, available at <http://www.wirelessinfo.com/content/palm-Treo-750-Cell-Phone-Review/Messaging.htm>, Mar. 17, 2007, 6 pages.
Palay et al., “The Andrew Toolkit: an Overview”, Information Technology Center, Carnegie-Mellon University, 1988, pp. 1-15.
“Palm User Guide”, 2005-2007, 755 pages.
Palm, Inc., “User Guide: Your Palm Treo.tm. 755p Smartphone”, 2005-2007, 304 pages.
Patterson et al., “Rendezvous: an Architecture for Synchronous Multi-user Applications”, CSCW '90 Proceedings, 1990, pp. 317-328.
Pearl, Amy, “System Support for Integrated Desktop Video Conferencing”, Sunmicrosystems Laboratories, Dec. 1992, pp. 1-15.
Phillips, Dick, “The Multi-media Workstation”, Siggraph '89 Panel Proceedings, 1989, pp. 93-109.
Phoenix Solutions, Inc., “Declaration of Christopher Schmandt Regarding the MIT Galaxy System”, West Interactive Corp., a Delaware Corporation, Document 40, Jul. 2, 2010, 162 pages.
Raper, Larry K., “The C-MU PC Server Project”, (CMU-ITC-86-051), Dec. 1986, pp. 1-30.
Reexam Non-Final Office Action received for U.S. Appl. No. 90/013,163, dated Aug. 1, 2014, 45 pages.
Request for Ex Parte Reexamination, received for U.S. Appl. No. 90/013,163, filed Feb. 25, 2014, 265 pages.
Rice et al., “Monthly Program: Nov. 14, 1995”, The San Francisco Bay Area Chapter of ACM SIGCHI, Available at <http://www.baychi.org/calendar/19951114>, Nov. 14, 1995, 2 pages.
Rice et al., “Using the Web Instead of a Window System”, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI'96, 1996, pp. 1-14.
Rivlin et al., “Maestro: Conductor of Multimedia Analysis Technologies”, SRI International, 1999, 7 pages.
Roddy et al., “Communication and Collaboration in a Landscape of B2B eMarketplaces”, VerticalNet Solutions, A Business White Paper, Jun. 15, 2000, 23 pages.
Roddy et al., “Interface Issues in Text Based Chat Rooms”, SIGCHI Bulletin, vol. 30, No. 2, Apr. 1998, pp. 119-123.
Root, Robert, “Design of a Multi-media Vehicle for Social Browsing”, Bell Communications Research, 1988, pp. 25-38.
Rosenberg et al., “An Overview of the Andrew Message System”, Information Technology Center Carnegie-Mellon University, Jul. 1987, pp. 99-108.
Scheifler, R. W., “The X Window System”, MIT Laboratory for Computer Science and Gettys, Jim Digital Equipment Corporation and MIT Project Athena; ACM Transactions on Graphics, vol. 5, No. 2, Apr. 1986, pp. 79-109.
Schnelle, Dirk, “Context Aware Voice User Interfaces for Workflow Support”, Dissertation paper, Aug. 27, 2007, 254 pages.
Schooler et al., “A Packet-switched Multimedia Conferencing System”, By Eve Schooler, et al; ACM SIGOIS Bulletin, vol. I, No. 1, Jan. 1989, pp. 12-22.
Schooler et al., “An Architecture for Multimedia Connection Management”, Proceedings IEEE 4th Comsoc International Workshop on Multimedia Communications, Apr. 1992, pp. 271-274.
Schooler et al., “Multimedia Conferencing: Has It Come of Age?”, Proceedings 24th Hawaii International Conference on System Sciences, vol. 3, Jan. 1991, pp. 707-716.
Schooler et al., “The Connection Control Protocol: Architecture Overview”, USC/Information Sciences Institute, Jan. 28, 1992, pp. 1-6.
Schooler, Eve M., “Case Study: Multimedia Conference Control in a Packet-switched Teleconferencing System”, Journal of Internetworking: Research and Experience, vol. 4, No. 2, Jun. 1993, pp. 99-120.
Schooler, Eve M., “The Impact of Scaling on a Multimedia Connection Architecture”, Journal of Multimedia Systems, vol. 1, No. 1, Aug. 1993, 11 pages.
Schooler, Eve, “A Distributed Architecture for Multimedia Conference Control”, ISI Research Report, Nov. 1991, pp. 1-18.
Seneff et al., “A New Restaurant Guide Conversational System: Issues in Rapid Prototyping for Specialized Domains”, retrived from Internet on Oct. 1996 <citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.16.. rep . . . ,>, Oct. 1996, 4 pages.
Sheth et al., “Relationships at the Heart of Semantic Web: Modeling, Discovering, and Exploiting Complex Semantic Relationships”, Enhancing the Power of the Internet: Studies in Fuzziness and Soft Computing, Oct. 13, 2002, pp. 1-38.
Simonite, Tom, “One Easy Way to Make Siri Smarter”, Technology Review, Oct. 18, 2011, 2 pages.
Smith et al., “Guidelines for Designing User Interface Software”, User Lab, Inc., Aug. 1986, pp. 1-384.
Srinivas et al., “Monet: a Multi-media System for Conferencing and Application Sharing in Distributed Systems”, CERC Technical Report Series Research Note, Feb. 1992, 19 pages.
Stent et al., “The Commandtalk Spoken Dialogue System”, SRI International, 1999, pp. 183-190.
Supplemental Non-Final Office Action received for U.S. Appl. No. 11/848,208, dated Apr. 20, 2011, 15 pages.
Textndrive, “Text'ndrive App Demo—Listen and Reply to Your Messages by Voice while Driving!”, YouTube Video available at <http://www.youtube.com/watch?v=WaGfzoHsAMw>, Apr. 27, 2010, 1 page.
Tidwell, Jenifer, “Animated Transition”, Designing Interfaces, Patterns for effective Interaction Design, First Edition, Nov. 2005, 4 pages.
Tofel Kevin C. et al., “Speaktoit: a Personal Assistant for Older iPhones, iPads”, Apple News, Tips and Reviews, Feb. 9, 2012, 7 pages.
Touch, Joseph, “Zoned Analog Personal Teleconferencing”, USC / Information Sciences Institute, 1993, pp. 1-19.
Trigg et al., “Hypertext Habitats: Experiences of Writers in Notecards”, Hypertext '87 Papers; Intelligent Systems Laboratory, Xerox Palo Alto Research Center, 1987, pp. 89-108.
Trowbridge, David, “Using Andrew for Development of Educational Applications”, Center for Design of Educational Computing, Carnegie-Mellon University (CMU-ITC-85-065), Jun. 2, 1985, pp. 1-6.
Tucker, Joshua, “Too Lazy to Grab Your TV Remote? Use Siri Instead”, Engadget, Nov. 30, 2011, pp. 1-8.
Tur et al., “The CALO Meeting Assistant System”, IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, No. 6, Aug. 2010, pp. 1601-1611.
Tur et al., “The CALO Meeting Speech Recognition and Understanding System”, Proc. IEEE Spoken Language Technology Workshop, 2008, 4 pages.
Turletti, Thierry, “The INRIA Videoconferencing System (IVS)”, Oct. 1994, pp. 1-7.
Viegas et al., “Chat Circles”, SIGCHI Conference on Human Factors in Computing Systems, May 15-20, 1999, pp. 9-16.
Vlingo et al., “Vlingo Launches Voice Enablement Application on Apple App Store”, Press Release, Dec. 3, 2008, 2 pages.
Vlingo Lncar, “Distracted Driving Solution with Vlingo Incar”, YouTube Video, Available at <http://www.youtube.com/watch?v=Vqs8XfXxgz4>, Oct. 2010, 2 pages.
Voiceassist, “Send Text, Listen to and Send E-mail by Voice”, YouTube Video available at <http://www.youtube.com/watch?v=0tEU61nHHA4>, Jul. 30, 2009, 1 page.
VoiceontheGo, “Voice on the Go (blackberry)”, YouTube Video, available at <http://www.youtube.com/watch?v=pJqpWgQS98w>, Jul. 27, 2009, 1 page.
Wadlow, M. G., “The Role of Human Interface Guidelines in the Design of Multimedia Applications”, Carnegie Mellon University (to be Published in Current Psychology: Research and Reviews, Summer 1990 (CMU-ITC-91-101), 1990, pp. 1-22.
Walker et al., “The Locus Distributed Operating System 1”, University of California Los Angeles, 1983, p. 49-70.
Watabe et al., “Distributed Multiparty Desktop Conferencing System: Mermaid”, CSCW 90 Proceedings, Oct. 1990, pp. 27-38.
Wirelessinfo, “SMS/MMS Ease of Use (8.0)”, available at <http://www.wirelessinfo.com/content/palm-Treo-750-Cell-Phone-Review/Messaging.htm>, Mar. 2007, 3 pages.
Zelig, “A Review of the Palm Treo 750v”, available at <http://www.mtekk.com.au/Articles/tabid/54/articleType/ArticleView/articleId/769/A-Review-of-the-Palm-Treo-750v.aspx>, Feb. 5, 2007, 3 pages.
Ziegler, K, “A Distributed Information System Study”, IBM Systems Journal, vol. 18, No. 3, 1979, pp. 374-401.
Zue, Victor W., “Toward Systems that Understand Spoken Language”, ARPA Strategic Computing Institute, Feb. 1994, 9 pages.
Zue, Victor, “Conversational Interfaces: Advances and Challenges”, Spoken Language System Group, Sep. 1997, 10 pages.
Related Publications (1)
Number Date Country
20190363999 A1 Nov 2019 US
Divisions (1)
Number Date Country
Parent 10428523 May 2003 US
Child 11688664 US
Continuations (3)
Number Date Country
Parent 15714783 Sep 2017 US
Child 16504713 US
Parent 13908998 Jun 2013 US
Child 15714783 US
Parent 11688664 Mar 2007 US
Child 13908998 US