Patent Application
20130139103
The present disclosure relates generally to displaying applications, which are executing remotely on a server, on a mobile device. In particular, the present disclosure is directed to integrating native user interface components on a mobile device with a virtualized application.
With the pervasiveness of computers, network access, and Internet usage in daily personal and professional activities, many users desire mobility and flexibility. As a result of the mobility offered by various mobile devices, such as smart phones and tablets, users typically seek convenience and familiarity in their computing needs. Many applications include graphical user interfaces that were developed for personal computers and laptops with large displays and conventional input devices, such as a keyboard and mouse. Further, a user may be used to interacting with an application through desktop operating systems, such as the Windows operating system of Microsoft Corp. and the Macintosh Operating System (Mac OS) of Apple Inc. However, mobile devices are often designed to be compact and have limited display and input capabilities. Further, the user interface(s) and/or operating system(s) installed on a mobile device are often designed with the compact nature, limited display and limited input capabilities of the mobile device in mind, and, thus, provide a user with a different experience than the desktop operating systems. Although several methods have been developed to present desktop applications on mobile devices, the results typically range from cumbersome and clumsy to not feasible or cost-prohibitive. Thus, there remains an ever present need to improve the presentation of desktop applications on mobile devices and to improve a user's experience when interacting with desktop applications that are being presented on mobile devices.
The following presents a simplified summary of the disclosure in order to provide a basic understanding of some aspects. It is not intended to identify key or critical elements of the disclosure or to delineate the scope thereof. The following summary merely presents some concepts of the disclosure in a simplified form as a prelude to the more detailed description provided below.
By enabling mobile devices, such as smart phones and tablets, to leverage native user interface components, the methods and systems described herein provide users a more seamless experience, wherein the user can potentially be unaware of the fact that the application is not executing locally on the mobile device, but instead is executing remotely. For example, a server may be hosting the application and may be in communication with the mobile device to enable the mobile device to present the application, such as a window for the application in a graphical user interface of the mobile device. In some embodiments, the systems and methods described herein provide a user interface module to trigger the display of a native user interface component, such as a smartphone touch screen keyboard, in connection with the window for the application. As one example, whenever a text input field on the server gains or loses user focus, a remote computing device may send a message to the mobile device. The message can be interpreted by the mobile device to trigger or dismiss the native user interface component.
Additionally, in some embodiments, the systems and methods described herein auto-adjust the pan and zoom settings on the mobile device to ensure that windows for the application are presented in a manner that makes it easier for the user to complete various interactions. For example, using windowing information, a user can zoom into different regions of a window for the application including a zoom out and a zoom in.
In some embodiments, the server may be hosting the application in a remote desktop and as windows pop-up or get created in the remote desktop, the windows might not be visible on the mobile device. The systems and methods described herein permit the user to switch to or bring in focus the new window. In some embodiments, a visual cue may be displayed to indicate that a window has been created somewhere on the remote desktop.
The details of these and other embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims.
The present disclosure is illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:
Some embodiments may be described in the context of a network providing services to user devices over various networks using one or more protocols. The disclosure is not limited to any particular type of network or networks using a specific type of communication medium or to a specific set of communication protocols.
As also illustrated in
Although shown as a single network in
Devices 105-120 may be configured to interact with each other or other devices, such as remote machine 125 and remote machine 130. In one example, mobile device 110 may include client software 165 that is configured to coordinate the transmission and reception of information to and from remote machine 130. In one arrangement, client software 165 may include application or server specific protocols for maintaining connections to and requesting/receiving data from remote machines 125 and/or 130. For example, client software 165 may comprise a Web browser or mobile variants thereof and content provider/server 130 may comprise a web server. Client software 165 may be stored in computer-readable memory 160 such as read only, random access memory, writeable and rewriteable media and removable media in device 110 and may include instructions that cause one or more components—for example, processor 155, a transceiver, and a display—of device 110 to perform various functions and methods including those described herein.
In some embodiments, the network environment 100 may include multiple, logically-grouped remote computing devices. In one of these embodiments, the logical group of remote machines may be referred to as a server farm. In another of embodiment, the remote computing devices may be geographically dispersed (e.g., physically located in different geographic regions, countries, states, etc.). In some variations, a server farm may be administered as a single entity.
Each of the remote computing devices may be a file server, application server, web server, proxy server, gateway, application gateway, gateway server, or the like. Additionally, remote computing devices may be one or more blade servers or virtual machines. In some embodiments, remote machines 125 and 130 may provide a remote authentication dial-in user service, such as RADIUS. In one or more variations, the remote computing devices may have the capacity to function as either an application server or as a master application server.
The remote computing devices, in some embodiments, can be any server type. In other embodiments, the remote computing devices can be any of the following server types: a file server; an application server; a web server; a proxy server; an appliance; a network appliance; a gateway; an application gateway; a gateway server; a virtualization server; a deployment server; a SSL VPN server; a firewall; a web server; an application server or as a master application server; a server executing an active directory; or a server executing an application acceleration program that provides firewall functionality, application functionality, or load balancing functionality. In embodiments where a remote computing device comprises an appliance, the appliance may be from any one of the following manufacturers, among others: the Citrix Application Networking Group; Silver Peak Systems, Inc; Riverbed Technology, Inc.; F5 Networks, Inc.; or Juniper Networks, Inc.
In some embodiments, remote machine 125 and remote machine 130 may provide local computing devices (e.g., device 105, 110, 120 and 115) access to a computing environment. A computing environment may comprise an application, a plurality of applications, a desktop. desktop environment, and/or a desktop session in which one or more applications may execute. In some instances, remote machine 125 or 130 may execute an application on behalf of a user of a local computing device. Remote machine 125 or 130 may execute a virtual machine, which provides an execution session within which applications execute on behalf of a user of a local computing device. On some arrangements, the virtual machine may be managed by a hypervisor, such as the Xen hypervisor, developed and sold by Citrix Systems, Inc.; Hyper-V, developed and sold by Microsoft Corp.; ESX hypervisor developed and sold by EMC, or any other hypervisor. Communication between the remote computing devices and local computing devices may be performed so that local computing devices may request execution of various applications hosted by the remote computing devices and/or that local computing devices may receive data related to the application for presentation (e.g., display) on the local computing devices.
As used herein, a desktop, desktop environment, or desktop session refers to a graphical environment or space in which one or more applications may be hosted and/or executed. A desktop may include a graphical shell providing a user interface for an instance of an operating system in which local and/or remote applications can be integrated. Applications, as used herein, are programs that execute after an instance of an operating system (and, optionally, also the desktop) has been loaded. Each instance of the operating system may be physical (e.g., one operating system per device) or virtual (e.g., many instances of an OS running on a single device). Each application may be executed on a local device, or executed on a remotely located device (e.g., remoted).
In some embodiments, each of the remote computing devices may be running one or more applications, such as an application providing a thin-client computing or remote display presentation application. In one embodiment, the remote computing devices may be executing as an application any portion of the Citrix Access Suite by Citrix Systems, Inc., such as the Metaframe or Citrix Presentation Server products; any of the Microsoft Windows Terminal Services manufactured by the Microsoft Corporation; and/or any of the following products manufactured by Citrix Systems, Inc.: Citrix XenApp, Citrix XenDesktop, Citrix Access Gateway. Various other suitable applications can also be executed, including an Independent Computing Architecture (ICA) client, developed by Citrix Systems, Inc.; an application providing email services such as Microsoft Exchange manufactured by the Microsoft Corporation; a web or Internet server application; a desktop sharing server application; and a collaboration server application. Additionally or alternatively, the remote computing devices may provide various types of hosted services or products, such as GoToMeeting provided by Citrix Online Division, Inc.; WebEx provided by WebEx, Inc.; and Microsoft Office Live Meeting provided by Microsoft Corporation.
Each of the local computing devices may execute, operate or otherwise provide one or more applications to facilitate remote computing. Such applications can be of various type and/or form such as a web browser, web-based client, client-server application, a thin-client computing client, ActiveX control, Java applet, or the like. Using such applications, a local computing device may be able to provide remotely hosted applications on the local computing devices. In some arrangements, the remotely hosted applications may be a server-based or a remote-based application executed on behalf of the local computing device on a remote computing device (e.g., remote machine 125). In one or more embodiments, a remote computing device may communicate data for display to the local computing device using any thin-client protocol, presentation layer protocol, or remote-display protocol, such as the ICA protocol; the Remote Desktop Protocol (RDP); the X11 protocol; the Virtual Network Computing (VNC) protocol; the SPICE protocol; the Net2Display protocol; the PC-over-IP protocol; the TCX protocol; the THINC protocol; and the Virtual-D protocols. Applications provided by the local computing device may use any type of protocol and one or more of the applications provided by the local computing device can be, for example, an HTTP client, an FTP client, an Oscar client, a Telnet client, software related to voice over Internet protocol (VoIP) communications; and applications related to real-time data communications (e.g., software for streaming video and/or audio).
In some embodiments, the remote computing devices and local computing devices may be configured to communicate with each other via any type or form of gateway or tunneling protocol, such as Secure Socket Layer (SSL), Transport Layer Security (TLS), and Citrix Gateway Protocol of Citrix Systems.
Some embodiments include a first remote computing device that receives requests from a local computing device, forwards the request to a second remote computing device, and responds to the request generated by the local computing device with a response from the second remote computing device. The first computing device can acquire an enumeration of applications available to the local computing device and well as address information associated with the remote computing device that is hosting an application identified within the enumeration of applications. The first remote computing device can then present a response to the client's request using, for example, a web interface. In some arrangements, the first remote computing device may communicate directly with the local computing device to provide the local computing device with access to an identified application.
The
Mobile device 212 or its various components may be configured to receive, decode and process various types of transmissions including digital broadband broadcast transmissions that are based, for example, on the Digital Video Broadcast (DVB) standard, such as DVB-NGH, DVB-H, DVB-T2, DVB-H+ (hybrid satellite/terrestrial architecture), or Digital Video Broadcasting—Multimedia Home Platform (DVB-MHP), through a specific broadcast transceiver 241. Other digital transmission formats may alternatively be used to deliver content and information regarding availability of supplemental services. Additionally or alternatively, mobile device 212 may be configured to receive, decode and process transmissions through various transceivers, such as FM/AM Radio transceiver 242, wireless local area network (WLAN) transceiver 243, and telecommunications transceiver 244.
Although the above description of
Computing devices, such as the examples depicted in
One or more aspects of the disclosure may be embodied in computer-usable data and/or computer-executable instructions, such as in one or more program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types when executed by a processor in a computer or other data processing device. The computer executable instructions may be stored on one or more computer readable media such as a hard disk, optical disk, removable storage media, solid state memory, RAM, etc. The functionality of the program modules may be combined or distributed as desired in various embodiments. In addition, the functionality may be embodied in whole or in part in firmware or hardware equivalents such as integrated circuits, field programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), and the like. Particular data structures may be used to more effectively implement one or more aspects, and such data structures are contemplated within the scope of computer executable instructions and computer-usable data described herein.
One or more aspects of this disclosure relate to providing resources from a remote computing device to a local computing device. A resource may be a program, application, document, file, graphical window, or the like. In some variations the resource (also referred herein as a remote resource or hosted resource) may be operating in a desktop environment of the remote computing device. In one or more embodiments, a user may initiate a connection between a local computing device and a remote computing device. While connected, the user may interact with the remote computing device to request various actions for processing by the remote computing device, including, for example, execution of a resource, zooming in of a window, zooming out of a window, placing a window in focus, placing a window out of focus, termination of a resource, editing of a resource, and saving of a resource.
In some embodiments, remote computing device 300 may be executing a hosting agent 302. The hosting agent 302 may be configured to receive data from the local computing device 320, including requests for resources, interactions with the resources, among other various data communications. In some variations, the hosting agent 302 may be configured to send data to the local computing device 320, including data related to one or more hosted resources 304 being executed by the remote computing device 300. In some arrangements, hosting agent may be part of or in communication with a presentation layer protocol agent (now shown). The hosting agent 302 may include software, hardware or any combination thereof. Additionally, the hosting agent 302 may be configured to communicate with applications executing on the remote computing device 300 (e.g., hosted resources 304, which in some embodiments can be referred to as virtualized resources, virtualized applications, remote resources, or remote applications) and/or access data related to such applications (e.g., output data 308, window data 310). Further, the hosting agent 302 may use a presentation layer protocol (layer 6 of the OSI model) for at least some communications.
While the one or more hosted resources 304 are executing, each may generate output data 308 and window data 310. Output data 308 and window data 310 may include the data used to prepare, render or include in a graphical window for a hosted resource, including, for example, window attributes defining properties of the graphical window (e.g., window size, position, lifetime, Z-order), image data of the graphical window, data objects defining graphical elements that are displayed in the graphical window, data defining the window's relationship to another resource (e.g., an indication whether the window is a child of another window or application), object location data identifying locations for the graphical elements, data defining title bar contents for the graphical window, data describing the visual state of the graphical window (e.g., transparency), an icon (or bitmap of an icon) for the graphical window, window location data identifying the location for the graphical window in a desktop environment, various identifiers for the resource, and a path to the resource or the process that generated the resource. In some embodiments, the one or more hosted resources 304 may be executing within a desktop environment of the remote computing device 300. Additionally, in some variations, output and window data 308 may also include one or more commands, such as commands to create a window, delete a window, move a window, bring focus to a window, make a window transparent or partially transparent, open a menu of the window, close a menu of the window, resize a window, select a button or other element included in the window, etc.
Portions of output data 308 and window data 310 may be communicated to the local computing device 102 to enable the local computing device 320 to render a local version of a graphical window for the one or more hosted resources in local desktop environment 330, or otherwise graphically integrate the hosted resource into the local desktop environment 330. In some arrangements, hosting agent 302 may be configured to directly communicate output data 308 and window data 310 to the client agent 322 (e.g., via a port, channel or tunnel). Alternatively, hosting agent 302 may be configured to cause or initiate communication of the output data 308 and window data 310 to the local computing device 320.
Client management engine 322 may receive data from the remote computing device 300, including output data 308 and window data 310. In some arrangements, the data may be received via dedicated ports, channels or tunnels. For example, client agent 322 may receive output data 308 in one port, channel or tunnel, and may receive window data 310 for the hosted application in another port, channel or tunnel. In some arrangements, client agent 322 may receive the data using a dynamically linked library component that receives output data 308 and/or window data 310. In turn, the received data may be used in presenting a local version of the window on the graphical user interface 330.
More fully, client agent 322 may be configured to interface with operating system 326 of the local computing device to render a local version of the hosted resource. For example, client agent 322 may interact with a shell of the operating system. The client agent 322 may be configured to communicate with the operating system 326 to modify the configuration and/or display of the operating system's graphical user interface (e.g., desktop environment). For example, client agent 322 may be configured to store output data and window data received from remote computing device 300 and may call one or more functions from a shared library, such as a dynamically-linked library (DLL) to maintain or modify graphical user interface 330 (e.g., modify a taskbar, modify a placement of a window, close a window, create a window, modify contents of a window). Through the interactions with the operating system, the client agent 322 may provide a local version of a hosted resource, which is executed by remote computing device 300, to a user of the local computing device. The client agent 322 may be configured to modify various elements of the graphical user interface 330 The methods used to modify the elements of the graphical user interface 330 may depend on the operating system 326 operating on the local computing device 320. For example, a method for used when operating system 326 is a version of Android may be different than a method used when operating system 326 is a version of iOS.
At step 401, the mobile device (or client agent executing on the mobile device) may receive data related to a remote resource (e.g., output data and window data) from a remote computing device and present a window for the remote resource on a graphical user interface of the mobile device.
In some embodiments, the mobile device may interpret the data related to the remote resource and apply an automatic zoom or position the window for the remote resource to be visible to the user. In some embodiments, the user provides preferences for zoom percentage and events or data than trigger an automatic zoom. In some embodiments, the preferences are predetermined by a system administrator or at time of manufacture.
In some embodiments, secondary windows, new windows, or “child” windows may be presented as part of the remote resource. In some embodiments, as the secondary windows pop-up, the mobile device may apply a zoom to the secondary windows so that they are presented in a manner that makes interacting with the window easier. For example, if a “File Open” dialog box is launched by the remote resource, the mobile device may receive the information that a secondary window was created and can adjust the zoom and pan position to display the dialog to better fit on the display screen of the mobile device. In some embodiments, there is sufficient metadata in the presentation layer protocol to determine which windows should be automatically brought to focus and made visible to the user. In some embodiments, windows such as “Tooltips” can be discriminated against and won't be automatically panned or zoomed. Such automatic panning and zooming (and the discrimination against automatic panning or zooming) may be performed again upon display of the native user interface control, which is described below in connection with step 404, and, upon the native user interface component being removed from the display of the mobile device, which occurs at step 407.
In some embodiments, the data related to the remote resource that was received from the remote computing device can be used to display visual cues of window activity in the remote resource even if only a portion of the remote desktop hosted by the remote computing device is displayed on the mobile device. In some embodiments, the visual cues enable the user of the mobile device to keep track of the various windows created on the remote desktop even if they are not displayed on the screen of the mobile device.
In some embodiments, the user will be able to jump between windows within a session. Using the automatic zoom and automatic positioning functionality, the user can seamlessly tab between various windows.
At step 402, the mobile device (or client agent executing on the mobile device) may receive information associated with the window of the remote resource from the remote computing device such as, for example, an initialization packet. In some instances, the initialization packet may be accompanied by the data related to a remote resource (e.g., the data received at step 401). In others, the initialization packet may be received in a transmission different from one that includes the data related to the remote resource. Further, in some arrangements, the initialization packet may be received as part of output or window data. Additionally, as will be discussed further below, the initialization packet may be received responsive to a click event being transmitted from the mobile device to the remote computing device.
At step 403, the mobile device (or client agent executing on the mobile device) may process the initialization packet to determine a native user interface component required by a remote resource. In some arrangements, the initialization packet may include various data types that are used to determine the native user interface component. For example, coordinate information, size information and the like that is included in the initialization packet may be used to identify the native user interface component (e.g., window size, window position, window lifetime, and/or Z-order).
In some embodiments, the information included in the initialization packet may be compared against window data of the mobile device's graphical user interface, or window data for the remote resource that was received from the remote computing device. Based on the comparison, the mobile device may determine a particular native interface control to activate. For example, if the comparison identifies a text input control, the native user interface component may be a keyboard native to the mobile device. If the comparison identifies a menu control, the native user interface component may be a drop down menu native to the mobile device (although, in some instances, the native drop down menu may be populated with information text received from the remote computing device). If the comparison identifies a date picker control, the native user interface component may be a date picker. Various other types of native user interface control may be determined, including any control that could be determined based on information such as window size, window position, window lifetime, Z-order, and the like. In some arrangements, the initialization packet may include a command from the remote computing device that identifies a particular native interface control to activate.
At step 404, the mobile device (or the client agent executing on the mobile device) may cause display of the native user interface component of the responsive to the determination.
At step 405, the mobile device (or the client agent executing on the mobile device) may receive a termination packet from the remote computing device.
At step 406, the mobile device (or the client agent executing on the mobile device) may process the termination packet to identify the native user interface component. The information included within a termination packet that is used to determine which native user interface component to terminate may be similar to the information used at step 403. For example, the termination packet may include coordinate information, size information and the like (e.g., window size, window position, window lifetime, and/or Z-order). In some arrangements, instead of receiving a termination packet at step 406, another initialization packet may be received. In such arrangements, the mobile device may determine the native user interface component from the initialization packet, determine that the native user interface component is being displayed on the mobile device, and in response, may proceed to step 407.
At step 407, the mobile device (or the client agent executing on the mobile device) may cause the native user interface component to be hidden in the graphical user interface of the mobile device, or may otherwise remove the native user interface component from the graphical user interface.
Native user interface components may be determined based on click events received at the mobile device.
At step 501, the mobile device (or client agent executing on the mobile device) may intercept a click event that was generated based on a user interaction with the graphical user interface of the mobile device.
At step 502, the mobile device (or client agent executing on the mobile device) may display a visual cue on a display of the mobile device responsive to the received click event. In some embodiments, the visual feedback persists until the click event is acted on by remote resource.
At optional step 503, the mobile device (or client agent executing on the mobile device) may transmit the click event to the remote computing device.
In some arrangements, transmitting the click event to the remote computing device may cause the remote computing device to transmit the information received at step 402 of
At optional step 504, the mobile device (or client agent executing on the mobile device) may receive a message or a packet from the remote computing device indicating that the click event initiated by the user on the mobile device was acted upon. In some embodiments, the packet received at step 504 may be a termination packet or an initialization packet, which can also cause the mobile device to remove the native user interface component to be not displayed on the mobile device (e.g., steps 406 and 407 of
At step 505, the mobile device (or client agent executing on the mobile device) may hide the visual cue responsive to resolution of the received client event.
Identification of specific commercial hardware and/or software products are for illustrative purposes only, and are not meant to be limiting in any way. Other hardware and/or software with similar or equivalent functionality may also be used. Additionally, numerous other embodiments, modifications and variations within the scope and spirit of the appended claims will occur to persons of ordinary skill in the art from a review of this disclosure.
This application is a non-provisional of and claims priority to co-pending U.S. provisional application Ser. No. 61/564,517, filed Nov. 29, 2011, entitled “METHODS AND SYSTEMS FOR INTEGRATING NATIVE USER INTERFACE COMPONENTS ON A MOBILE DEVICE WITH A VIRTUALIZED APPLICATION,” herein incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 61564517 | Nov 2011 | US |
