Patent Application
20120226992
The present invention relates to the field of user interfaces. More specifically, the present invention relates to a remote user interface media adapter in a network bridge.
The number of electronic devices in people's homes is continually increasing. Many years ago, homes only had a radio; then, a radio and a television. The number of devices has increased to the point where a typical home has several televisions, stereos, computers, video game consoles, mobile phones/devices, appliances and others. Furthermore, these devices are gaining intelligence so that they are able to communicate with each other.
The expansion of residential networks to include a multiplicity of devices that can share files asynchronously and connect to the Internet through residential gateways was facilitated by the de-facto standard use of wired and wireless ethernet connectivity. Asynchronous sharing then started to give way to buffered streaming of video as bandwidth availability improved. This was closely followed by real time streaming. Networks employ quality of service to manage bandwidth resources and Universal Plug and Play (UPnP) to perform discovery and compatibility of compressed video content. Video UPnP also defines remote user input operation like play, stop and rewind so that video control as well as video display is able to be performed remotely. Also, provisions were made to support graphical transfer of a remote user interface, but no implementations on the market have made use of this. UPnP allowed for many different standards of compressed video, but does not, however, certify that a client supported the relevant decoder. Digital Living Network Alliance (DLNA) is a standards body formed to provide certified device compatibility for a specific subset of UPnP implementations. DLNA also defined the role of media servers, renderers, adapters, players and controllers.
A standard, referred to as Remote User Interface (RUI or Remote UI) is being developed to allow devices to operate each other and provide the user with a user interface that is configured appropriately for a device being used to control another device. For example, a user interface for a 46″ wide television is not likely to appear properly on a mobile phone which has a display of 2″. The Remote UI standard is a web-based protocol and framework for remote user interface on UPnP Networks and the Internet. The standard allows a UPnP-capable home network device to provide its interface (display and control options) as a web page to display on any other device coupled to the home network.
There are no well defined and widely accepted UPnP implementations for graphical RUI. One option, which has been backed by the UPnP Forum, is a browser based implementation known as CEA2014. The network client browser is considered to be heavy in flash, memory and/or processor requirements (‘thick’ client), whereas the network server application performs simple encapsulation of XML (‘thin’ server). In some situations this may be acceptable, like the case when rendering is performed by a personal computer and the application is run on a small mobile device, or a low end processing device, like a network router.
However, in the case of the home network where the rendering is done by a high definition TV, a Blu-Ray® player, a picture frame or a gaming machine, the use of a browser for RUI has some disadvantages. Firstly, a browser adds to the already substantial memory requirements of the renderers and so for these cost sensitive consumer electronics devices it may not be viable. Secondly, the processing speed requirements for a responsive experience are not going to be provided by the current range of devices available. And thirdly, the browser interface lends itself well to mouse and keyboard control, but is not necessarily the ideal format for a limited button remote control.
Also, the home network is able to include graphics applications built into game machines, video players, dongles and intelligent remotes on the low end, with cable boxes, cloud servers and multimedia PCs on the high end. To shoehorn all of these into one UPnP standard, it is clear that reach will be limited. In some cases substantial effort of rewriting or translation of the graphics application might be needed in order to fit the browser framework.
Another example of a proposed RUI is being provided through the RVU alliance. The RVU alliance was initiated by DirectTV in order to provide a pixel accurate remotely rendered version of their satellite decoder user interface. Unlike the browser based RUI, RVU uses a low level protocol that manipulates the graphics card framebuffer layers more directly. Instead of the script type messages that CEA2014 uses, RVU breaks up elements of the graphics into images that can be sent compressed or uncompressed over the network to be composited in the renderers screen buffers or off screen buffers as needed. Simple bit commands are sent over the network to allow the images to be stretched, cut and alpha-blended on the renderer side. This type of RUI would be considered a thin network client and thick network server because most of the computation effort would be with the application. Also, because most actions involve sending image data, this type of RUI uses a lot of network resources.
The advantage of RVU is that the low level graphics operations are able to be supported by all graphics cards quite easily and is not directly dependent on the type of application to be able to function. However, sometimes performance is a key parameter in usability, and as such the network load and network server performance could severely limit how useful the protocol is. RVU is especially vulnerable where complete screen refreshes are needed often, like 3D rotations of a view. A browser approach could handle this more simply through scripts of simple rotation commands. Another similar limitation is when the application is providing remote graphics to multiple renderers, and causes the application processor to run short of the necessary MIPS to perform adequately.
A bridge including an adapter is able to transcode or translate different graphics languages to permit a variety of networked, dissimilar devices, to communicate. The bridge translates an existing application device's graphics/images into a Remote User Interface (Remote UI or RUI) graphics protocol of a rendering device and transmits the translated RUI to the rendering device. The rendering device is then able to display the translated RUI.
In one aspect, a method of implementing a bridge comprises detecting a rendering device on a network, presenting existence information of the rendering device to an application device, opening a session between the application device and the rendering device and communicating a remote user interface between the application device and the rendering device using the session. The method further comprises presenting transcoding capabilities to the application device. The method further comprises performing additional processing to lessen a processing load on at least one of the application device and the rendering device. Opening the session is initiated by the application device. Communicating further comprises translating a command in a first remote user interface protocol into the command in a second remote user interface protocol. Translating further comprises receiving the command in the first remote user interface protocol and sending the command in the second remote user interface protocol. The first remote user interface protocol is different from the second remote user interface protocol. The bridge is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPhone, an iPod®, a video player, a DVD writer/player, a television, a home entertainment system and an intelligent appliance.
In another aspect, a bridge device comprises a memory for storing an application, the application for detecting a rendering device on a network, presenting existence information of the rendering device to an application device and enabling communication of a remote user interface between the application device and the rendering device and a processing component coupled to the memory, the processing component for processing the application. The processor processes additional data received from at least one of the devices within the network. The application further presents transcoding capabilities to the application device. The application device and the rendering device implement disparate remote user interface protocols. The application receives a command in a first remote user interface protocol and sends the command in a second remote user interface protocol. An adapter in the bridge device monitors network traffic and application device/rendering device loading to make decisions about which type of remote user interface protocol to be used for optimal overall performance. The bridge device is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPhone, an iPod®, a video player, a DVD writer/player, a television, a home entertainment system and an intelligent appliance.
In another aspect, a network of devices comprises a first device implementing a first remote user interface protocol, a second device implementing a second remote user interface protocol and a bridge device implementing an adapter capable of communicating in the first protocol and the second protocol. The first device is an application device and the second device is a rendering device. The bridge device receives commands in the first remote user interface protocol from the first device and communicates the commands in the second remote user interface protocol to the second device. The first protocol and the second protocol are disparate protocols.
In yet another aspect, a system programmed in a controller in a device comprises a detecting module for detecting devices in a network; and an adapting module for translating a command in a first remote user interface protocol to the command in a second remote user interface protocol. The system further comprises a determining module for determining a protocol implemented by each of the devices. The device are selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPhone, an iPod®, a video player, a DVD writer/player, a television, a home entertainment system and an intelligent appliance.
A device and method that extends a multi-technology bridge or router with media adapter functionality for both a graphics Remote User Interface (RUI) and video services to maximize the bandwidth usage and universal interconnectivity in the home is described herein. The method includes detecting multiple types of network interconnects and is able to expand Bluetooth®. The media adapter is able to be any media adapter, for example, as described in U.S. Patent Application No. Attorney Docket No. Sony-39800, filed ***, titled “REMOTE USER INTERFACE ADAPTER,” which is incorporated by reference herein. Additionally, the U.S. Patent Application No. Attorney Docket No. Sony-39700, filed ***, titled, “TIERED HIERARCHICAL REMOTE USER INTERFACE,” is incorporated by reference herein.
In the past, network/processor bandwidth limitations and the state of technology excluded the possibility of transcoding video and translating a graphics RUI on the network for the purpose of distributing content to rendering devices that have different technologies from an application server device. With the bridge described herein, transcoding video and translating an RUI are able to be implemented. Multi-technology bridges' and routers' capabilities are maximized by embedding video transcoding and graphics RUI translating services into these devices.
A network discoverable graphics and video media adapter included with a network bridge or router allows applications to detect all potential target rendering devices on the network. The rendering device may not inherently support the video or graphics capability of the application device, therefore transcoding takes place. Then, for example, if the home supports different types of home networks, such as WiFi-bg 2.4 gHz for some devices, WiFi-N 5 gHz for some and ethernet for others, the maximum bandwidth and flexibility is able to be offered if the transcoding is done inside of the bridge. Generating a target market for rendering devices that reside in one WiFi spectrum or that use a different IP networking technology such as Multimedia over Coax Alliance (MoCA), maximizes the potential benefit of placing the transcoding capabilities within a network bridge.
A networked application device that resides on one home Internet Protocol (IP) network discovers the video and graphical transcoder capabilities embedded within the media adapter included in a network bridge or router that resides on that network. The media adapter presents information about all available rendering devices that are coupled on the network. The media adapter also presents transcoding capabilities. The application device then opens up a session with a rendering device, making use of transcoding capabilities of the bridge.
The application device 102 is any device which is able to be used to communicate data such as an RUI and RUI commands to rendering devices. For example, the application device 102 is a mobile phone which is being used as a remote control for a television.
The first rendering device 106 (e.g. a television) and the second rendering device (e.g. a Blu-ray® player) 108 are any devices that render data for the purpose of displaying the data.
The bridge device 104 is any device which is able to be used to couple network segments and analyze incoming data packets to determine if the packet is able to be sent to another segment of a network. The bridge device 104 is also able to be a link to a larger network such as the Internet. The bridge device 104 includes an adapter 110. The adapter 110 is able to be implemented in software, hardware, firmware or any combination thereof. In some embodiments, the bridge device 104 is discoverable by other devices on the network.
The adapter 110 within the bridge device 104 transcodes or translates different graphics languages to permit a variety of networked, dissimilar devices, to communicate by partially rendering or translating an existing application's devices' graphics/images into a lower level RUI graphics protocol; where the RUI graphics protocol is delivered over the network to a rendering device (e.g. a PC with XII and a television with 2014). In some embodiments, the adapter monitors network traffic and application device/rendering device loading to make intelligent decisions about which type of RUI protocol should be used for optimal overall performance.
The adapter 110 is able to implement middleware such as a Blu-ray® disc Java stack. Additionally, the adapter 110 provides an adapter implementation which communicates in many different RUI protocols and interfaces. For example, the adapter 110 is able to receive a command in one RUI protocol and then output the command in a second RUI protocol so that a device that communicates based on the second RUI protocol is able to understand the command. As a further example, if the application device 102 attempts to send a command to the first rendering device 106, but the application device 102 communicates using OpenVG and the first rendering device 106 only accepts commands in OpenGL, then, typically, the devices would not be able to communicate. However, with the bridge device 104 using the adapter 110, the application device 102 sends the command in OpenVG to the bridge device 104 which uses the adapter 110 and translates the command into OpenGL and sends an OpenGL command to the first rendering device 106 which is able to receive the OpenGL command. Although a command is described herein, any information is able to be sent, translated and the re-sent. In some embodiments, the bridge device 104 is able to perform rendering processing or other processing to help with the processing load on the network.
In some embodiments, the network of devices 100 includes a network. The network is able to be any type of network such as a Local Area Network (LAN), a Wide Area Network (WAN), the Internet, a network of networks or any other network. Additionally, the type of network is able to be wireless, wired, cellular, any other type of network or any combination of two or more networks.
Although the network of devices 100 of
In some embodiments, the remote user interface bridge application(s) 330 include several applications and/or modules. As described herein, a device detection module for detecting/discovering devices and an adapting module for translating/transcoding information or commands from one protocol to another protocol are implemented. In some embodiments, there are additional modules or sub-modules such as a presenting module for presenting detection information, a transcoding capability module for presenting transcoding capabilities and a session module for opening a session between application devices and rendering devices. In some embodiments, modules include one or more sub-modules as well. In some embodiments, fewer or additional modules are able to be included.
Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPod®/iPhone, a video player, a DVD writer/player, a Blu-ray® writer/player, a television, a home entertainment system or any other suitable computing device. In some embodiments, a computing device is able to include intelligent appliances such as a refrigerator, a toaster, a toaster oven and a microwave, where the appliances are able to process and/or present information.
To utilize the bridge, a user incorporates the bridge in a network by coupling the bridge to the devices in the network. The coupling is able to be wired, wireless or a combination thereof. Depending on the implementation, the bridge determines the protocols implemented by the devices on the network and performs the adapting (also referred to as translating) from one protocol to another, so that commands are able to be transmitted/received by devices with similar or disparate protocols. The determining and adapting is able to occur automatically or manually depending on the implementation. Although it has been generally described that a command is translated into another protocol, the command is able to be translated into several protocols. For example, if a user wants to turn on many devices, but each device implements a different RUI protocol than a controlling device, the bridge is able to translate the command into many different protocols at the same time so that the devices all turn on. From a user's perspective, the user generally does not notice that the bridge is functioning aside from disparate devices being able to communicate and devices functioning more efficiently with the processor load being shared with the bridge.
In operation, the bridge determines protocols implemented by devices being communicated with. If the devices implement the same protocol, then the bridge passes any communications sent from one device to the other device. However, if the devices implement disparate protocols, the bridge is able to translate from one protocol to another so that the devices are able to communicate. The bridge is also able to process commands or perform computations to lighten a processing load of another device such as a rendering device.
The devices implemented within the network described herein are able to implement Digital Living Network Alliance (DLNA) standard as well.
1. A method of implementing a bridge comprising:
a. detecting a rendering device on a network;
b. presenting existence information of the rendering device to an application device;
c. opening a session between the application device and the rendering device; and
d. communicating a remote user interface between the application device and the rendering device using the session.
2. The method of clause 1 further comprising presenting transcoding capabilities to the application device.
3. The method of clause 1 further comprising performing additional processing to lessen a processing load on at least one of the application device and the rendering device.
4. The method of clause 1 wherein opening the session is initiated by the application device.
5. The method of clause 1 wherein communicating further comprises translating a command in a first remote user interface protocol into the command in a second remote user interface protocol.
6. The method of clause 5 wherein translating further comprises receiving the command in the first remote user interface protocol and sending the command in the second remote user interface protocol.
7. The method of clause 6 wherein the first remote user interface protocol is different from the second remote user interface protocol.
8. The method of clause 1 wherein the bridge is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPhone, an iPod®, a video player, a DVD writer/player, a television, a home entertainment system and an intelligent appliance.
9. A bridge device comprising:
a. a memory for storing an application, the application for:
b. a processing component coupled to the memory, the processing component for processing the application.
10. The bridge device of clause 9 wherein the processor processes additional data received from at least one of the devices within the network.
11. The bridge device of clause 9 wherein the application further presents transcoding capabilities to the application device.
12. The bridge device of clause 9 wherein the application device and the rendering device implement disparate remote user interface protocols.
13. The bridge device of clause 9 wherein the application receives a command in a first remote user interface protocol and sends the command in a second remote user interface protocol.
14. The bridge device of clause 9 wherein an adapter in the bridge device monitors network traffic and application device/rendering device loading to make decisions about which type of remote user interface protocol to be used for optimal overall performance.
15. The bridge device of clause 9 wherein the bridge device is selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPhone, an iPod®, a video player, a DVD writer/player, a television, a home entertainment system and an intelligent appliance.
16. A network of devices comprising:
a. a first device implementing a first remote user interface protocol;
b. a second device implementing a second remote user interface protocol; and
c. a bridge device implementing an adapter capable of communicating in the first protocol and the second protocol.
17. The network of devices of clause 16 wherein the first device is an application device and the second device is a rendering device.
18. The network of devices of clause 16 wherein the bridge device receives commands in the first remote user interface protocol from the first device and communicates the commands in the second remote user interface protocol to the second device.
19. The network of devices of clause 16 wherein the first protocol and the second protocol are disparate protocols.
20. A system programmed in a controller in a device comprising:
a. a detecting module for detecting devices in a network; and
b. an adapting module for translating a command in a first remote user interface protocol to the command in a second remote user interface protocol.
21. The system of clause 20 further comprising a determining module for determining a protocol implemented by each of the devices.
22. The system of clause 20 wherein the device are selected from the group consisting of a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, an iPhone, an iPod®, a video player, a DVD writer/player, a television, a home entertainment system and an intelligent appliance.
The present invention has been described in terms of specific embodiments incorporating details to facilitate the understanding of principles of construction and operation of the invention. Such reference herein to specific embodiments and details thereof is not intended to limit the scope of the claims appended hereto. It will be readily apparent to one skilled in the art that other various modifications may be made in the embodiment chosen for illustration without departing from the spirit and scope of the invention as defined by the claims.
