Patent Application
20110200035
The background of this invention relates to how an external entity, such as a software application, another device, or other computer system can remotely can co-operatively control the user interface elements of a telephony capable end user device, such as a VoIP phone.
It addresses the rudimentary implementation and significant limitations of existing prior art such as found in US Patent Application 20090154678.
Voice over Internet Protocol (VoIP) devices and systems are now commonplace. Most end-user VoIP devices will have a user interface that typically includes elements such as a screen, possibly a touch screen, physical buttons and/or virtual buttons on a touch screen, and illuminated indicators such as LEDs. The devices typically present a command DTMF dial pad, along win non-dialpad keys.
There are many instances where it is desirable for external devices or systems to have the capability to control the user interface of a VoIP device. The purpose is to present a customizable user interface, without the need to modify the operating software resident in the VoIP device. Examples are if a VoIP phone would like to navigate through an address book that might be resident on a phone system server, or would like to present a visual list of voicemails available on the server. Once the customized information is presented on the VoIP device, user input on the VoIP device allows a user to navigate a list of information generated on the fly, and make appropriate action selections. This presentation of customized information can be anything to left to one's imagination, including the ability to view weather reports, stock information, etc. . . .
Many VoIP phones can accomplish this today via use of a HTTP-based XML browser resident in the VoIP phone. Content is requested by a VoIP phone, via a HTTP request, and content, commonly in XML or web page format is rendered and delivered by an external system.
This HTTP method provides acceptable in many situations, but has several drawbacks in many other situations. For simple content delivery it is sufficient, but in many cases integrators want to deliver customized content based on phone and end user specific events. This action events could be such items as pressing a feature key, going off hook, going on hook, putting a call on hold, accepting a second call, receiving an incoming call, initiating an outgoing call, upon a call being connected, when the phone is in an idle state, and other like actions. To provide this type of user interface control via the HTTP method, the VoIP phone software requires deep modification to hook in the various points where the VoIP phone will need execute a HTTP request. This type of software modification is intrusive and difficult to implement and not well standardized by various VoIP phone vendors.
This HTTP method is completely independent from a VoIP phones underlying control protocol, which is commonly the Session Initiation Protocol (SIP). In a SIP session, where media can be real-time audio and/or video media streams, it becomes very difficult to make the HTTP method leverage any integration to do with the media streams.
Take for instance, take the example of a user of a VoIP phone, who wants to review voicemails stored on a central phone system server. Using HTTP methods, it is common to see implementations to allow a user to navigate and review a list of voicemails present on the central phone system server. The user presses a button, which initiates a HTTP request to a central phone system server. The server returns a formatted list of entries. The user selects an entry, thereby initiating another HTTP request, and then the central phone server or VoIP phone initiates a SIP call between themselves to allow the user to listen to the actual voicemail via a SIP session audio media stream.
The above voicemail scenario is somewhat acceptable, but has many limitations. There is a significant amount of HTTP and SIP activity to perform this, which burdens both software developers of the VoIP device, and software developers on the central phone system server. The co-ordination of two protocols, HTTP and SIP becomes cumbersome. Also, in situations where security and encryption is important, it would be necessary to encrypt both protocols, incurring more software overhead.
The invention described herein, provides a more flexible customized content control, but using just the SIP protocol. When a SIP call is initiated, and once connected, then media, be it audio and/or video, are established between two SIP devices. As known to one skilled in the art, dial pad key presses are relayed back to the other SIP device via packets encoded with DTMF information.
Now we will look at this invention, with the same voicemail scenario example. A user of a VoIP phone initiates a voice call, via SIP protocol, to his central voicemail server to listen to his voicemails, as is normally done when calling in to listen to voicemail messages on a central Internet Protocol based central phone system server. Once the SIP call session is established, audio, and possibly video streams flow between the user of the VoIP phone, and the central phone system server. Typically, the user is required to enter DTMF dialpad entries to navigate audibly through a list of voicemails. There is no textual or graphical representation of voicemail entries presented when the SIP call session is established.
The invention described herein, allows the central phone system server to send special user interface control (UI-CTL) SIP messages on the same dialog as the established SIP session. Various UI-CTL SIP message(s) sent from the central phone system server allow textual or graphical content to be displayed on the end user VoIP device. This way, a selection of voicemail entries can be presented. A very important innovation though, is that in these UI-CTL SIP messages, one can specify DTMF keys to be sent upon the end user of the VoIP phone pressing keys other than the regular DTMF dialpad keys. This is important to improve the end user experience, because many end user VoIP devices have many other non-dialpad keys, such as left, right, up, down navigation keys, softkeys, virtual keys displayed on a touchscreen, and various other keys. These UI-CTL SIP messages are simple messages that convey display information that completely or partially display on the screen in a co-operative manner on the end user VoIP device.
The power of this customization of DTMF responses can be seen in contrast to a prior art example, such as US Patent Application 20090154678. In the 20090154678 application, the end user of the VoIP device is restricted to only entering in regular DTMF dialpad entries. That severely limits the friendliness of the user interface, and occupies a lot of precious screen area just to indicate which DTMF key entries will be acted upon. In the 20090154678 application
For the invention described herein, and in reference to the above voicemail example, the central phone server sends UI-CTL SIP messages on the same dialog as the established media SIP session. This UI-CTL SIP message can specify what DTMF keys are to be sent upon an end user of the VoIP phone pressing various non-dialpad keys on the VoIP phone. For example, if a VoIP phone device has four navigation keys (left, right, up, down) and an OK button, the UI-CTL SIP message could comprise information that tells the VoIP phone device to send one or more specific DTMF key values for each of these non-dialpad keys. With this innovation, a much richer user interface experience is possible, since the end user is not limited to pressing just regular DTMF dialpad entries. One or more UI-CTL SIP messages can also describe textual or graphical information that is rendered by the end user VoIP device.
In reference to the prior art US Patent Application 20090154678, it is also important note that these UI-CTL messages are sent in-dialog with the existing SIP session. This is important, because a VoIP device may be involved in a multitude of SIP sessions simultaneously. An end user of a VoIP device may put one session on hold, and accept another new session. The remote entities may be sending UI-CTL messages, and the SIP dialog information in the UI-CTL will allow the application on the end user VoIP device to know what SIP communication session the UI-CTL messages are associated with. Application 20090154678 only allows one session at a time, and does not allow any method for the end-user to manage multiple VoIP sessions simultaneously.
In reference to prior art, Application 20090154678 indicates that it is the callee who is generating the user interface control messages. The invention described in this document allows either the caller or callee, or both to be actively sending user interface control messages.
In reference to prior art, Application 20090154678 also does not provide any ability for the user interface control messages to control other user interface elements on a VoIP device such as LEDs. The invention described in this document allows control of indicators such as LEDs, or graphical elements that may be rendered on a display screen.
In contrast to the HTTP method used on many VoIP phones, the inventive methods allow much simpler implementation of a flexible user interface control, and allow tight integration with media streams such as audio or video. This dramatically simplifies the implementations both on the end user display device, and the device, application or system generating the user interface content.
The present disclosure is directed to overcoming one or more of the problems associated with allowing co-operative user interface control of an end user VoIP device.
Reference will now be made in detail to the exemplary preferred embodiments implemented according to the disclosure, the examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.
Exemplary embodiments include a method and system for implementation of an inventive system and methods to allow another device to cooperatively control the user interface elements of a second device. Exemplary embodiments include devices communicating over an Internet Protocol (IP) network using the Session Initiation Protocol (SIP) communication protocol between the two devices. The SIP protocol is well known to one skilled in the art, and IETF RFC3261 is one of the RFC documents that describe this protocol. Each device incorporates a SIP user agent, as is known to one skilled in the art. When the devices are idle, or are in the process of establishing a SIP communication session, or are already in an established SIP communication session, either device's SIP user agent can send user-interface control (UI-CTL) messages to the other device's SIP user agent.
Upon a SIP user agent receiving a UI-CTL message, the recipient SIP user agent can decide to act upon the message to modify any operation in its user interface. The UI-CTL message can control what textual or graphical information that is displayed on a device's screen, can modify what action is taken when a recipient's keys, whether real, or virtual as shown on a touchscreen, are subsequently pressed, and can modify the display characteristics indicators, whether real or virtual on the recipient device screen. These indicators would include such things as LEDs on a device, or a virtual indicator rendered on a screen. The display characteristics would cover such items as color, blinking cadences that would be visually noticed by a user.
User Interface Device 302-1 (and 302-2) represents an electronic device that has some level of user interface on it, which may include such elements as a screen, buttons and indicators such as LEDs.
To one skilled in the art, it is recognized that a User Interface Device 302-1 could be realized in many different manners, and the above description is just a representative example. For example, an alternative representation of User Interface Device 302-1 can be found in
In the inventive system, at least one of the SIP User Agents 301 must be part of a User Interface Device 302. The other end of the SIP communication could be another User Interface Device 302, or it could be an entity which has no user interface, such as an Application 303.
An Application 303 would represent an entity that does not have any user interface, but rather would be a software application that communicates to the User Interface Device 302-1 via its own SIP User Agent 301-2. Typical examples of an Application 303 known to one skilled in the art would include a PC application, and private branch exchange server (PBX), an interactive voice response (IVR) system, a voicemail system, a generic embedded electronic system, and other similar systems that have been designed with an embedded SIP User Agent 301.
The SIP User Agents 301 communicate using the Session Initiation Protocol (SIP) protocol, but one skilled in the art will appreciate that the disclosure could be implemented using other VoIP protocols, e.g., H.323 or other packet-signalling VoIP telephony protocols known in the art.
After the SIP communication session is established, SIP User Agent 301-2 can now send a standard SIP NOTIFY message, shown in
In a preferred embodiment the SIP call flow diagram 500, it is recommended that no SUBSCRIBE SIP message is required from SIP User Agent 301-1 to initiate the reception of UI_CTL_NOTIFY( ) 340 message. The reason for this is that this UI-CTL session is established upon each SIP call, and a SUBSCRIBE would add an unnecessary amount of overhead. But it is imperative that the UI_CTL_NOTIFY( ) 340 message is sent, as is known to one skilled in the art, “in dialog” to the original INVITE( ) 320 message transaction. The reason for this is that Device 302-1 could be managing multiple SIP User Agents, such as SIP User Agent 301-1 at the same time, and hence that Device 302-1 may be managing multiple simultaneous user interface sessions. By having the UI_CTL_NOTIFY( ) 340 message “in dialog” to the original INVITE( ) 320 message transaction allows Device 302-1 to associate the UI_CTL_NOTIFY( ) 340 to the proper SIP call session. It is understood by one skilled in the art, that if the Application 300 is unable to send the UI_CTL_NOTIFY( ) 340 message “in dialog”, the less preferred method, would be for Application 300 to send the UI_CTL_NOTIFY( ) 340 message “out of dialog”. But in this case, the content body of the UI_CTL_NOTIFY( ) 340 message would need to additionally convey the appropriate SIP session dialog identification information. This identification information would include appropriate dialog identification information such as the SIP To, From, tags, Call-ID header fields. This would allows Device 302-1 to match this “out of dialog” UI_CTL_NOTIFY( ) 340 message to the appropriate SIP call session.
Upon reception and acceptance of UI_CTL_NOTIFY( ) 340 message by SIP User Agent 301-1, it can now parse the Content Body of the UI_CTL_NOTIFY( ) 340 message, and render this UI information properly on Device 302-1. The following is an exemplary example of the UI_CTL_NOTIFY( ) 340 message, including its Content Body. Note that this information could be sent across multiple UI_CTL_NOTIFY( ) 340 messages, and not all information elements need to be the same for each UI_CTL_NOTIFY( ) 340 message. For explanation purposes, it is shown as a single message. As an example, this UI_CTL_NOTIFY( ) 340 message could be rendered Device UI Interface 300 as shown in
The following will provide a description of the purpose of each XML element in this Content Body.
Element: <root></root>
This simply represents the end and the start of the XML body, for parsing purposes. The screen=overlay parameter indicates this is a full screen overlay over the default connected SIP session screen that Device 302-1 would normally render. For example, screen=update may be used to update a portion of the screen.
The <Text> element represents a textual string to be rendered on the screen 116, along with the X and Y location co-ordinates on the screen. In this examples, character-based display icons, which are device/implementation specific can be supported. $BOOK$ represents a book character-based icon.
This represents a line to be drawn on the screen, along with the starting X and Y location co-ordinates on the screen, it's pixel length, and pixel height.
This represents that a LED indicator is to be have its operation changed. <Item> specifies which LED. In this case F1 may represent the first feature key LED, as represented in
This represents a key inventive concept in the operation of Device 302-1. Devices as shown in
For example, <Key>SOFTKEY_BL, DTMF_RFC2833,4</Key> would indicate that when the bottom left softkey is pressed, that Device 302-1 would send DTMF value of 4, using the DTMF RFC2833 method, as is known to one skilled in the art. Note that in this example, more than one DTMF digit could be specified, and they are can even include extended DTMF values, such as A,B,C,D as is known to one skilled in the art.
It is quite common for DTMF digits to be conveyed by the RFC2833 method, and that is normally negotiated between two SIP User Agents when the SIP communication is established. Known to one skilled in the art, these would be negotiated to use RFC2833, INFO or in-band methods in SIP INVITE( ) 320 and 200 OK( ) 322 messages. The <Key> action entry allows Application 303 to effectively override what DTMF method is used for dialpad keys. Any other keys not specified in <Keys> would continue their default behaviour on Device 302-1.
Note that Application 303 can send a new UI_CTL_NOTIFY( ) 340 message at any time. In many cases it would send a new UI_CTL_NOTIFY( ) 340 message based upon receiving a DTMF event from SIP User Agent 301-1.
Referencing the previously full SIP message format of the UI_CTL_NOTIFY( ) 340 message, one can now envision how key presses, both dialpad, and non-dialpad keys would be reported back to Application 303. For example, referencing
Note that the during a call session, if Application 303 desires Device 302-1 to return to its default screen 116 presentation, the preferred method of doing this is to send a UI_CTL_NOTIFY( ) 340 message with an empty Content Body. Having this functionality would allow Application 303 to implement momentary screen pops on Device 302-1.
Note that the <Key> identifier in the UI_CTL_NOTIFY( ) 340 message can incorporate actions beyond just DTMF key presses. The defined actions can be expanded to include a multitude of imaginative actions that are executed solely by Device 302-1, or include actions that initiate other SIP protocol requests such as BYE, INVITE, REFER, NOTIFY, etc. . . .
For example, if Application 303 is an Asterisk PBX implementation, as is known to one skilled in the art, the Asterisk PBX may want to send a UI_CTL_NOTIFY( ) 340 message to Device 302-1 to add a Call Park softkey to Screen 116 of Device 302-1. Note that for an Asterisk PBX, for Device 302-1 to park a call, Device 302-1 is required to perform an attended transfer SIP message sequence to a predefined call park extension number on the Asterisk PBX. For example,
In this exemplary example, this UI_CTL_NOTIFY( ) 340 message would not overwrite the complete Screen 116, but would rather add a Call Park softkey to the upper right softkey of Device 302-1. When a user presses this upper right softkey on Device 302-1, it would initiate an attended SIP call transfer to the SIP address 700@192.168.0.50, as is known to one skilled in the art. This is a powerful innovation that allows tremendous flexibility for Application 303, in this example, acting as an Asterisk PBX, to extend the default user interface functionality of Device 302-1.
The extensibility of these <Key> functions provide a powerful capability for Application 303 to cooperatively control the user interface Device 302-1, all using the standard SIP communications protocol. This is contrasted to other systems where either the user interface on Device 302-1 cannot be controlled, or if there is some user interface control, it is implemented with a hybrid of other TCP/IP protocols such as HTTP. Coordinating a user interface experience can be difficult for application developers when there is a multitude of TCP/IP networking protocols in use. The methods of our invention is a powerful inventive concept for application developers who want to extend the functionality of user interface devices such as Device 302-1, using the same SIP protocol used to set up the call session. The ability to tightly couple the user interface experience, in synchronicity with the SIP messaging and RTP streams is highly desirable.
It is noted that when Application 303 receives DTMF key events from Device 302-1, that in addition to sending additional UI_CTL_NOTIFY( ) 340 messages, Application 303 can also manipulate the audio and/or video RTP payloads being delivered to and from Device 302-1. This can be useful a variety of different implementations of Application 303.
For example, if Application 303 acts as a voicemail server, to one skilled in the art, it can now be seen how a visual voicemail application can be delivered to Device 302-1. A user of Device 302-1 can make a SIP call to the voicemail application, and when connected, the voicemail application would send UI_CTL_NOTIFY( ) 340 messages to Device 302-1, and respond to DTMF events sent by Device 302-1. For example, if the voicemail server renders a list of voicemail entries, a selection scheme, and finally a “Play” softkey on Device 302-1, and the user of Device 302-1 presses a “Play” softkey, the voicemail server can start streaming a new RTP stream from an audio file stored on the voicemail server. Conversely, to one skilled in the art, the voicemail server can present a user interface to Device 302-1 that would allow simple message forward and reverse options to be presented on the Screen 116 of Device 302-1. Once these forward or reverse key options are pressed on Device 302-1, the voicemail server would forward or reverse its RTP streaming position of the stored audio file.
The above examples are a representation of some of the possible UI control actions that could be supported. Those skilled in the art will appreciate that this may be implemented in a variety of ways other than the XML description above, and may not cover off all possible UI actions that are possible. In particular, a more efficient XML schema may be deployed, in situations where a system would want to fit this UI_CTL_NOTIFY( ) 340 message into one TCP/IP UDP packet. In addition, other SIP message types, and or fields could be used, or other TCP/IP packet protocols than SIP could be used. In addition, elements such as <Display> can be enhanced to support display modifications of rich screen presentations, such as those rendered in HTML. As an alternative example, the <Display> attributes could be enhanced to allow modify any Document Object Model (DOM) element within a rich HTML screen presentation. The original HTML screen content can be resident in Device 302-1, or it could be requested via HTTP protocols, or finally, delivered to Device 302-1 via various enhancements to the UI_CTL_NOTIFY( ) 340 message.
The next inventive concepts for remote control of a device user interface in Internet telephony SIP environment 400 is the fact that the user interface of Device 302-1 can also be controlled outside of a connected SIP communication session.
For example, if Application 303 is an Asterisk PBX implementation, as is known to one skilled in the art, the Asterisk PBX may want to send a UI_CTL_NOTIFY( ) 340 message to Device 302-1 during this provisional to add a Voicemail redirection softkey to Screen 116 of Device 302-1. Note that for an Asterisk PBX, for Device 302-1 to redirect an incoming SIP call to voicemail, Device 302-1 is required to send a SIP redirect message to a predefined SIP address on the Asterisk PBX. For example,
In this exemplary example, this UI_CTL_NOTIFY( ) 340 message would not overwrite the complete Screen 116, but would rather add a “to Voicemail” softkey to the upper right softkey of Device 302-1. When a user presses this upper right key on Device 302-1, it would send a SIP 302 Redirect 345 message to the SIP address *200@192.168.0.50. Known to one skilled in the art, this has the effect of redirecting the incoming SIP call to SIP User Agent 301-1 back to the voicemail SIP address of the Asterisk PBX.
In a similar manner, as shown in
It is noted that not all <Key> action items can be performed on Device 302-1 when in the provisional or idle state. For example, known to one skilled in the art, no DTMF action or call transfer events can be initiated between SIP User Agents 301-1 and 301-2 when they are in idle or provisional state. But a SIP INVITE action event would be possible when Device 302-1 is in an idle state.
For Application 303 to know the user interface characteristics of Device 302-1, or to know if Device 302-1 even supports the inventive user interface control techniques, some method of discovery of the user interface control capabilities is needed. In the preferred embodiment of the invention, a UI_CTL_NOTIFY( ) 340 message is sent with an action request. This action request would indicate that it would like to discover the user interface control capabilities. As mentioned previously, if the Device 302-1 would reply with a standard SIP 4xx error response code if it did not support this capability at all. But if Device 302-1 does support the capability, in the preferred embodiment of the invention, it can reply back with the standard 200 OK SIP 341 message, and include in the content body of the 200 OK SIP 341 message any such informative information about its user interface control capabilities. This may include information such as screen characteristics, key and indicator availability information, and any necessary information for Application 303 to understand what user interface control capabilities. Known to one skilled in the art, there are other less preferred embodiments of this discovery capability, such as looking at the SIP User Agent field in the 200 OK 341 message. Known to one skilled in the art, the SIP User Agent field provides a textual name of the SIP User Agent 301-1. This SIP User Agent field would not normally have any user interface control information, but may identify the brand or model name of Device 302-1. From this information, Application 303 may be able to deduce the user interface control capabilities of Device 302-1.
It is noted that this cooperative external control of a device user interface can pose security risks, since the Application 303 could convey sensitive information. For example, if Application 303 is a banking application interactive voice response, it may be requesting to the user of Device 302-1, such as login information. Known to one skilled in the art, it is recommended that the SIP and associated RTP communication protocols have common industry security techniques applied to these protocols.
As is known to one skilled in the art, there is provided a computer usable medium for use in a device such as Device 302-1 and Application 303, with both having computer readable program code means embodied therein, the computer readable program code means for implementing their respective operations in the inventive cooperative external control of device user interface invention.
This application claims the benefit of U.S. Provisional Application No. 61295701, COOPERATIVE EXTERNAL CONTROL OF DEVICE USER INTERFACE USING SIP PROTOCOL, filed Jan. 16, 2010, which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61295701 | Jan 2010 | US |
