Patent Application
20120284632
The present disclosure relates generally to virtualized desktop environments and more particularly to providing an integrated rendering of media such as streaming media in a browser on a client endpoint device.
Web browsing is an increasingly popular activity in business and personal settings, and with the growth of network-connected devices such as personal computers, web-capable mobile phones and tablets has come increased demand for the provision of media over the web. For example, users may desire to conduct web-based audio and video conferencing, buy or rent movies or television shows over the web, view video or animation encoded for Adobe Flash, listen to streaming radio stations, or even play games with users around the world via the Internet.
When virtual or cloud-based desktops are used, web browsing may be virtualized along with other hosted applications. That is, a browser application may run in a hosted virtual desktop (HVD), or run as a hosted virtual application (HVA) while the browser window is displayed to a user on a remote client endpoint device such as a computer or mobile phone. Virtualized browsing presents a set of unique problems in that media such as streaming media may be more difficult to virtualize than simple text and graphics.
Overview
Techniques are provided for establishing an integrated rendering of a browser window comprising user interface elements such as streaming media on a client endpoint device. A web browser on a hosted virtual desktop (HVD) generates an HVD display image comprising a browser window and communicates it to the client endpoint device for display, via a virtual desktop interface (VDI) protocol. The browser window comprises a host-provided window element and a placeholder where client-provided data associated with a tag may be rendered. A client plugin server on the client endpoint device instantiates an endpoint browser plugin to render a tag in place of the placeholder portion of the HVD display, before displaying the integrated display of the browser window and rendered tag content at the client endpoint device.
Additional techniques are provided herein for rendering a web page comprising page content and a tag in a web browser on a hosted virtual desktop HVD, instantiating a stub plugin in the web browser, causing the stub plugin to render a placeholder into a portion of the browser window, establishing a plugin protocol session between the stub plugin and a plugin server on a client endpoint device, and sending information controlling the instantiation and operation of an endpoint plugin via the plugin protocol session, so that the client endpoint device can display a composited window of the web page.
Referring now to the Figures, an example of a block diagram showing a VDI environment in which VDI connectivity can be established between client endpoint devices and one or more hosted virtual desktops is shown in
Network 10 represents any hardware and/or software configured to communicate information via any suitable communications media (e.g., WAN, LAN, Internet, Intranet, wired, wireless, etc.), and may include routers, hubs, switches, gateways, or any other suitable components in any suitable form or arrangement. The various components of the VDI environment 100 may include any conventional or other communications devices to communicate over the networks via any conventional or other protocols, and may utilize any type of connection (e.g., wired, wireless, etc.) for access to the network.
Web server 20 is a conventional or other server for serving web pages including Hypertext Markup Language (HTML) documents and other content such as images or style sheets to the web browser 320. Content source servers 30a, 30b are conventional or other servers for serving data to a client or a content distribution cache server, e.g., a Darwin Streaming Server, Flash Media Server, Unreal Media Server, or the like. The content servers may provide any type of data, for example media such as streaming video and/or streaming audio, games or simulations, scripts, or the like. Content cache servers 35a-b, e.g. Cisco Wide Area Application Engine (WAE) servers running the Application and Content Network System (ACNS), act as intermediate repositories for content received from content servers 30a-b. As is further described with respect to
Host device 105 comprises one or more processors 110, a network interface unit 120, and memory 130. The processor 110 is, for example, a data processing device such as a microprocessor, microcontroller, system on a chip (SOC), or other fixed or programmable logic, that executes instructions for process logic stored in memory 130. The network interface unit 120 enables communication throughout the VDI environment, as shown in
The host device 105 may be, for example, a computing blade, a blade server comprising one or more solid state drives, or a blade center comprising one or more blade servers together with a blade chassis comprising common resources such as networking connections, input/output device connections, power connections, cooling devices, switches, etc. The host device 105 may be a component of a larger system, such as a Cisco Unified Computing System, or a data center that centralizes enterprise computing resources.
Resident in memory 130 are hypervisor 140, and multiple hosted virtual desktops (HVDs) 150a-d. The hypervisor or virtual machine monitor 140 presents a virtual operating platform to the HVDs 150a-d, and manages access to the host processor 110, network interface unit 120, memory 130 and other host resources, so that the HVDs 150a-d have access to appropriate host resources without disrupting each other's operation. Each HVD 150 operates independently of the other HVDs 150 and runs as a separate virtual machine on the host device 105, and each HVD 150 may run a different operating system if desired. Further operation of the HVDs is explained below with reference to
Each example client endpoint device 205a comprises one or more processors 210, a network interface unit 220, memory 230, and display rendering hardware 240. The processor 210 is, for example, a data processing device such as a microprocessor, microcontroller, system on a chip (SOC), or other fixed or programmable logic, that executes instructions for process logic stored in memory 230. The network interface unit 220 enables communication throughout the VDI environment, as shown in
The example client endpoint device 205 may be any conventional or other computer system or device, such as a thin client, computer terminal or workstation, personal desktop computer, laptop or netbook, tablet, cellular phone, set-top box, networked television, or other device capable of acting as a client in the described VDI environment.
The example client endpoint device 205 interfaces with display device 250, input device(s) 260, and output device(s) 270, and communicates with these devices in any suitable fashion, e.g., via a wired or wireless connection. The display device 250 may be any suitable display, screen or monitor capable of displaying information to a user of a client endpoint device, for example the screen of a tablet or the monitor attached to a computer workstation. Input device(s) 260 may include any suitable input device, for example, a keyboard, mouse, trackpad, touch input tablet, touch screen, camera, microphone, remote control, speech synthesizer, or the like. Output device(s) 270 may include any suitable output device, for example, a speaker, headphone, sound output port, or the like. The display device 250, input device(s) 260 and output device(s) 270 may be separate devices, e.g., a monitor used in conjunction with a microphone and speakers, or may be combined, e.g., a touchscreen that is a display and an input device, or a headset that is both an input (e.g., via the microphone) and output (e.g., via the speakers) device.
The functions of the processors 110 and 210 may each be implemented by a processor or computer readable tangible (non-transitory) medium encoded with instructions or by logic encoded in one or more tangible media (e.g., embedded logic such as an application specific integrated circuit (ASIC), digital signal processor (DSP) instructions, software that is executed by a processor, etc.), wherein the memories 130 and 230 each store data used for the computations or functions described herein (and/or to store software or processor instructions that are executed to carry out the computations or functions described herein). Alternatively, one or more computer readable storage media are provided and encoded with software comprising computer executable instructions and when the software is executed operable to performing the techniques described herein. Thus, functions of the process logic as described with reference to
The example HVD 150 comprises a VDI server 310; host operating system(s) 315; hosted web browser 320 further comprising HTML rendering engine 322, mapping database 330, host plugin 328, and stub plugin 324; and may also comprise one or more other application(s) 330. The example client endpoint device 205 comprises a VDI client 350, operating system(s) 355, and plugin server 360 (also called plugin element 360), which is connected via a client plugin application programming interface (API) 365 to endpoint plugin 370, all of which reside in memory 230 (as shown in
The VDI server 310 interacts with the host operating system 315 to provide virtual desktop interface functionality to the client endpoint device 205 over VDI session 405, which is a VDI protocol link that is established using any suitable VDI protocol, for example Citrix Independent Computing Architecture (ICA), VMWare PC over IP (PCoIP), Microsoft Remote Desktop Protocol (RDP), or other suitable protocol. For example, any application with which a user of the client endpoint device 205 is interacting is hosted by the HVD 150, while the window associated with the application is rendered by the client endpoint device 205. The windows are depicted and further described with reference to
The VDI client 350 interacts with client operating system 355, plugin server 360 and endpoint plugin 370 to render the received HVD display for display on the client endpoint device 205. As will be further described with reference to
After it receives the user input, VDI server 310 translates it into virtual keyboard and mouse inputs, and feeds it via host operating system 315 to host web browser 320 or another application 335, as if the applications and the input devices 260 were running on a single desktop computing device. The user inputs are processed by the appropriate application at the HVD, and HVD display images are generated by the operating system 315 and VDI server 310 for transmission back to the VDI client 350, which renders the HVD display and client-generated user elements for display to the user on display 250.
In another embodiment, host device 105 may execute hosted virtual applications (HVAs) from its memory 130, rather than full hosted virtual desktops 150. In this embodiment, client endpoint device 205 may use its VDI client 350 to interact with multiple host devices 105, each executing one or more HVAs, and use the client operating system 350 to composite the output of the HVAs to present a full windowed desktop on display 250.
The host web browser 320 may be any browser software capable of use in conjunction with the host operating system 315, for example Mozilla Firefox, Google Chrome, Microsoft Internet Explorer, Opera Software Opera, Apple Safari, etc., and it comprises an HTML rendering engine 322. When a user of client endpoint device 205 navigates to a web page in a displayed browser window using, e.g., a web address such as a Uniform Resource Identifier (URI), the HTML rendering engine 322 requests and receives from HTML server 20, for example, an HTML- or XHTML-encoded web page associated with the URI, over HTTP session 410. The web pages may contain a reference to an object that cannot be decoded natively by the HTML rendering engine 322, for example a tag such as an <object> or <embed> tag whose URI references content server 30, or whose Multipurpose Internet Mail Extension (MIME) type indicates a type of object (e.g., audio, video, Java, etc.) that cannot be decoded natively, or for a particular embodiment is not desired to be decoded natively. When the host web browser 320 encounters such an object, it refers to mapping database 330 to find an entry describing the plugin to render the object, and, if environment 100 is configured to execute and and/or render that object type on client endpoint device 205, it then instantiates the stub plugin 324 in the host web browser 320, and communicates with the stub plugin 324 via plugin API 326. This plugin API 326 is a bidirectional API, allowing the rendering engine 322 to make requests of the stub plugin 324, while also allowing the stub plugin 324 to signal events to the rendering engine 322 via a callback mechanism.
It is understood that web browser 320 will instantiate stub plugins only for those object types for which local execution on client endpoint device 205 is desired in a particular implementation. For all other object types, referral to mapping database 330 yields a host plugin 328 for that object type, which is instantiated and executed on the HVD 150. It is understood that in some implementations a particular object type may be executed on the host (via host plugin 328), whereas in other implementations the same object type may be associated with a stub plugin 324 for execution on the client endpoint device 205.
After instantiation, stub plugin 324 establishes a plugin protocol session 415 with the plugin server 360 resident on the client endpoint device 205. The plugin protocol session 415 may be established using any suitable protocol, for example HTTP, TLS, TCP, or any other suitable protocol. In one embodiment plugin protocol session is multiplexed into a virtual channel transported by VDI session 405. The plugin protocol comprises methods to identify the type of endpoint plugin 370 to be instantiated, to describe the location of one or more placeholder objects into which the endpoint plugin 370 should render its data and interact with the user, to identify a URI describing the location of the content server 30, 35 associated with a particular tag, and to transport application programming interface (API) requests between web browser 320 and endpoint plugin 370. The API requests may be specific to a browser or class of browsers and may support interfaces, for example, for Netscape Plugin API (NPAPI) for Mozilla Firefox and Seamonkey, Apple Safari, Google Chrome, and Opera Software Opera browsers; the Pepper Plugin API (PPAPI), for Google Chrome and open source Chromium browser; or the ActiveX API, for Microsoft Internet Explorer. These interfaces may be bidirectional, i.e., web browser 320 may make requests of endpoint plugin 370, and endpoint plugin 370 may make requests of web browser 320. A set of remote procedure calls (RPCs) may be used for communication of the APIs over this session 415 In one embodiment, the plugin protocol session 415 may be transported as a virtual channel within the VDI session 405, and in another embodiment the plugin protocol session 415 may be transported independently.
The plugin server 360 instantiates endpoint plugin 370 in response to interactions with the stub plugin 324 over plugin protocol session 415, and communicates with endpoint plugin 370 via a client plugin API 365. This plugin API 365 is a bidirectional API, allowing the plugin server 360 to make requests of the endpoint plugin 370, while also allowing the endpoint plugin 370 to signal events to the plugin server 360 via a callback mechanism. The plugin server 360 may be, for example, a software module or an element of a software module, and may be, for example, a stand-alone module, a part of another software module associated with client endpoint device 205, or a combination of both.
The endpoint plugin 370 is a browser plugin that is designed to render or interact with one or more MIME types that are not able to, or not desired to, be decoded natively by an HTML rendering engine 322, and which could not be rendered efficiently by a plugin executing on the HVD 150, for example a video plugin. Depending on the host and client operating systems 315, 355, a suitable plugin to be used as endpoint plugin 370 may be available “off-the-shelf” for use, or a plugin may need to be ported to the client operating system 355. In certain embodiments it is desirable to run an off-the-shelf plugin as endpoint plugin 370, in order to minimize development costs, simplify software distribution from existing repositories, and maximize the number of plugins for various MIME types that can be supported on the client endpoint device 205. However, in order to use an off-the-shelf plugin without rewriting it, the plugin server 360 and the client operating system 355 should provide an endpoint plugin API 365 and operating system 355 that is compatible with (e.g., the same as) the API expected by the off-the-shelf endpoint plugin 370.
The host browser 320 operates in conjunction with the endpoint plugin 370 to display the non-native object to the user of client endpoint device 205. When HTML rendering engine 322 calls a procedure in plugin API 326, stub plugin 324 converts that procedure call and its parameters to, for example, a remote procedure call (RPC) in the plugin protocol 415. When plugin server 360 receives such an RPC, it converts it to a procedure call on the client plugin API 365 to the endpoint plugin 370. Similarly, if endpoint plugin 370 makes a callback to plugin server 360, plugin server 360 may generate an RPC over plugin protocol session 415, which is in turn received by stub plugin 324, which converts it to a callback of plugin API 326 to HTML rendering engine 322. It is appreciated, therefore, that the plugin API 326 and the client plugin API 365 should be compatible.
In the example, responsive to commands made by client plugin API 365, endpoint plugin 370 establishes a content transport session 420 directly with content server 30. It is understood that content server 30 could also be a content cache server 35a-b with no substantial difference in the rest of the example. Thus, the content (e.g., media) data flows directly to client endpoint device 205, rather than flowing through the HVD 150 and thus requiring a very high bitrate from the VDI session 405. When the endpoint plugin 370 decodes and renders the data, the rendered data is sent to client operating system 355 to be merged with the rest of the HVD display, which is being rendered by VDI client 350. The data may be encoded or compressed in any suitable fashion, and transmitted via any suitable protocol, for example HTTP, Microsoft Media Services (MMS), MPEG-Transport Stream (MPEG-TS), the Real-time Transport Protocol (RTP), User Datagram Protocol (UDP), or any other suitable protocol.
As can be seen from
As compared to conventional methods that route content such as media from content servers through the HVD and over a VDI session to the client endpoint, the present embodiments use the content transport session 420 to directly transport content data to the client endpoint 205. This direct transportation of content to client endpoint devices has several benefits. First, using the content transport session 420 consumes less network bandwidth because it can maintain the native encoding of the content server, rather than forcing it to be transcoded to conform to the encoding used by the VDI session 405. Second, use of the content transport session 420 allows for Quality of Service (QoS) differentiation between regular VDI services and content delivery services. Third, transmitting content data directly to the client endpoints avoids needless concentration of bandwidth at a centralized location such as a host device 105 where multiple HVDs may be located. Fourth, using the content transport session 420 avoids placing high computing loads (e.g., media decode/encode loads) on the HVD, and thus avoids scalability problems on the HVD devices. Fifth, because the VDI session 405, HTTP session 410, plugin protocol session 415, and content transport sessions 420 are separate from each other, different network paths may be used for VDI communication, remote procedure calls, and content transmission. Sixth, the transport of content directly to the client endpoint devices allows efficient usage of cache server topology to reduce overall bandwidth across the network.
The various operating systems mentioned with reference to
The tag corresponding to the content data, will of course differ depending on the page fetched from web server 20, and comprises a MIME attribute that specifies a MIME type for the tag. Most conventional browsers can process an <object> or <embed> tag having, e.g., a MIME type such as application, audio, model, or video. There are numerous subtypes with these MIME types, for example, the application type includes hundreds of subtypes, e.g., for Flash, Silverlight, etc., and the video type includes dozens of subtypes, e.g., CCTV, H264, mp4, QuickTime, etc. A full list of MIME types (also known as internet media types) and subtypes is available from the Internet Corporation for Assigned Names and Numbers, also known as ICANN, at their website. In a preferred embodiment the tag has a MIME type of application or video, and in another preferred embodiment the tag has a MIME subtype of Flash, H264, JavaScript, mp4, Quicktime, RealPlayer, Shockwave, Silverlight, or Windows Media Player. In another preferred embodiment, the tag has a MIME type indicating telephony, video conferencing, or web-based push-to-talk. In yet another preferred embodiment, the MIME type indicates a game or simulation.
Although the description herein refers to a single endpoint plugin 370 for rendering the tag, it is understood that multiple endpoint plugins 370 may be instantiated, of the same or differing types, while displaying and interacting with a single web page. It is understood that, in the case where multiple endpoint plugins 370, either of the same or different types, are instantiated on the client endpoint device 205, a single type of stub plugin 324 can accommodate all of the MIME types to be supported, and that a single plugin server 360 can similarly accommodate as many types of endpoint plugins as are deemed appropriate for a particular embodiment. It is also understood that not all plugin types are appropriate for local rendering, and that any mixture of plugins hosted on both the host virtual machine 150 and the client endpoint device 205 can be supported.
In particular,
In the depicted example, the display element 535 is a plugin display window 535 filled with the media data rendered by the endpoint plugin 370, which in the depicted example is video data (e.g., CCTV, H264, mp4, QuickTime, etc.), but may also be any other type of data, such as Flash, JavaScript, or Silverlight. Furthermore, users may interact directly with the display plugin window using endpoint input devices such as a mouse or keyboard, rather than interacting with the HVD through the VDI session 405. Such interaction may occur when it is determined that the display plugin window has been granted focus, i.e. when the operating system determines that user input should be directed at a process in which the plugin is executing.
Although the depicted examples are of visual display elements, it will be understood that a similar compositing process takes place for audio. Client endpoint device 205 may receive audio, comprising, for example, application tones, music, or voice alerts, from HVD 150, via VDI session 405. Client endpoint device 205 may also receive audio content from a content server 30/35, via content transport session 420. Client endpoint device 205 should combine the audio from these two sources and render a coherent audio waveform to speakers 270. The two sources may, for example, be mixed by operating system 355, using audio rendering hardware in client endpoint device 205.
In particular, it is the responsibility of the client operating system 355 to accomplish compositing. In most windowed operating systems, compositing is accomplished by the operating system drawing each individual window according to a z-order, which describes the relative depth of the various windows. Windows with the deepest (lowest) z-order are drawn first, and then each window with a successively shallower (higher) z-order is drawn subsequently, which may result in the deeper windows being partially or fully occluded on the display. The assignment of higher-valued integers to shallower windows is somewhat arbitrary, but higher z-order shall herein be understood to imply shallower windows, i.e., windows stacked closer to the user's eyes.
It should be appreciated, however, that the VDI client 350 receives all virtual display information (i.e., the HVD display comprising browser window 510 with host-rendered element 520 and placeholder element 530, other HVD application windows 540, 550, and the HVD background desktop image 560) from VDI session 405 and requests the client operating system 355 to render the entire virtual display as a single rectangular window. Thus, although window 540 or 550 may have a higher z-order on the HVD than the browser window 510, the client endpoint device 205 (comprising, e.g., plugin server 360, endpoint plugin 370, and operating system 355) may composite the endpoint-rendered element 535 so that the composited images have a higher z-order than the HVD display.
The client endpoint device 205 creates the appearance that endpoint-rendered element 535 is partially occluded, however, by rendering either the endpoint-rendered element 535 or the remainder of display 505a as non-rectangular shapes. For example, the client endpoint device 205 may composite the endpoint-rendered element 535 in only the non-occluded portions of the placeholder element 530. This means that, for example as shown in
To efficiently render display 505a, the client operating system 355 should be able to accept requests to render non-rectangular images without interfering with the images on the rest of the display. The plugin server 360 should therefore be able to be informed of the non-rectangular geometry, so that this information may be communicated to the operating system 355. In one example, the stub plugin 324 interacts with HVD operating system 315 to compute the geometry of the non-occluded portions of the placeholder window and communicates that window geometry information to the plugin server 360 over the plugin protocol session 415. In another example, stub plugin 324 fills placeholder window 530 with a unique chromakey color, so that plugin server 360, endpoint plugin 370, or operating system 355, may compute the non-rectangular region by detecting what portions of the virtual display contain the chromakey color. Operating system 355 should also be able to render the non-rectangular images at high speed and at a low impact to the CPU of the client endpoint device 205. In one example of such an efficient rendering, operating system 355 is aware of display rendering hardware 240, which comprises, for example, a graphics processing unit (GPU) capable of rendering non-rectangular images.
In alternative
The initialization causes the stub plugin 324, in step 612, to establish a plugin protocol session 415 with plugin server 360. It will be understood that the transport of protocol session 415 may take many embodiments, including, but not limited to, use of HTTP, TLS, TCP, or multiplexing onto a virtual channel of VDI session 405. In step 614 stub plugin 324 sends an initialization remote procedure call (RPC), which contains both the initialization parameters from plugin API 326 and additional parameters indicating which type of endpoint plugin is to be loaded. The system is now initialized, and from this point onwards, events may be generated by user interactions with the web page, or scripting associated with the web page, or callback requests from the client endpoint device 205. At step 616, the stub plugin 324 waits for an event, and processes it according to a particular path before returning to step 616 to wait for another event.
At step 618, the stub plugin 324 receives a callback RPC from client endpoint device 205, and in step 620 the stub plugin 324 unmarshals (converts and disassembles) the RPC into parameters and generates a plugin API callback to the browser (e.g., HTML rendering engine 322) via plugin API 326. In step 622, the stub plugin 324 receives an API callback response, and at step 624 the callback results are marshaled (converted and assembled) into an RPC response, which the stub plugin 324 then sends back to the client endpoint device 205, via plugin protocol session 415, in step 626 before returning to step 616 to wait for another event.
At step 628, the stub plugin 324 receives a plugin API request from the browser (e.g., HTML rendering engine 322) via plugin API 326, and in step 630 the stub plugin marshals the parameters of the API request into an RPC request, and in step 632 sends the RPC request to client endpoint device 205 via plugin protocol session 415. In step 634, the stub plugin 324 receives an RPC response from the client endpoint device 205, unmarshals it and returns its results to the browser in step 636, before returning to step 616 to wait for another event.
At step 638, the stub plugin 324 detects a window position change, for example because the user is moving or resizing the windows in the displayed browser window 510 at the client endpoint 205, or because a host application needs to rearrange the windows, and in step 640 marshals the new window geometry, which may include information about what sections of the plugin window 535 are overlapped, into an RPC in step 640. The stub plugin 324 sends the RPC to the client endpoint device 205, via plugin protocol session 415, in step 642. In step 644, the stub plugin 324 receives an RPC confirmation from the client endpoint device 205 confirming that the RPC was processed, and then returns to step 616 to wait for another event.
At step 646, the stub plugin 324 receives a plugin API notification to unload the web page from the browser (e.g., HTML rendering engine 322) via plugin API 326, and in step 648 the stub plugin 324 creates an unload RPC and sends it to the client endpoint device 205, via plugin protocol session 415, in step 650. In step 652, the stub plugin 324 receives an RPC confirmation from the client endpoint device 205 confirming that the unload RPC was processed, and then the stub plugin 324 exits process 600 at step 654. At exit, the plugin protocol session 415 may or may not be terminated, depending on the particular embodiment. For example, if the user is still active, whether in the browser or another application, then the plugin protocol session 415 may be kept active until the browser is closed, the VDI session 405 is terminated, etc.
In step 702, the plugin server 360 waits until a plugin protocol session 415 is established. This step may entail verifying the credentials of the stub plugin 324 and responding to the session establishment request. It will be understood that the transport of protocol session 415 may take many embodiments, including, but not limited to, use of HTTP, TLS, TCP, or multiplexing onto a virtual channel of VDI session 405. Once the session is established, then in step 704 the plugin server 360 receives an initialization RPC request, which contains both the initialization parameters (e.g., window coordinates and size) from plugin API 326, and additional parameters (e.g., the MIME type) indicating which type of endpoint plugin is to be loaded, from the stub plugin 324. In step 706, the plugin server 360 uses the information in the initialization request, to locate and load the desired type of endpoint plugin 370.
At step 708, the plugin server 360 receives an HVD display image 500 from the HVD 150, where the HVD display comprises at least one client window element placeholder 530. The plugin server 360 may receive the HVD display image via VDI client 350 and the VDI session 405. The HVD display image may be transported as a single window comprising the entire HVD display image, or as a separate virtual image for each hosted virtual application, or a combination of the two.
In step 710, based on the initialization parameters, the plugin server 360 converts the drawing rectangle coordinates and size contained in the initialization RPC to coordinates suitable for the display 250 associated with client endpoint device 205. In step 712, the plugin server 360 creates a borderless window, that is, a drawing rectangle or a window with no framing decorations associated with it, for the plugin display window 535 as specified by the initialization. At this time, a list of regions occluding the drawing rectangle may also be received and processed. In sum, the drawing rectangle and occlusion information allows plugin server 360 to interact with operating system 355 to display only those portions of the data rendered by endpoint plugin 370 that are currently visible in the HVD display.
In step 714, the plugin server 360 initializes the endpoint plugin 370 via client plugin API 365 with the coordinates of the plugin display window 535 and other parameters from the initialization request, and, after waiting for the plugin to indicate that initialization was completed, in step 716 sends a response RPC to the stub plugin 324 indicating that initialization has completed. The system is now initialized, and from this point onwards, events may be generated by user interactions with the web page, or scripting associated with the web page, or requests from the stub plugin 324. At step 718, the plugin server 360 waits for an event, and at step 720 processes it according to a particular path before returning to step 718 to wait for another event.
At step 722, the plugin server 360 receives an API callback from the endpoint plugin 370, and in step 724 the plugin server 360 marshals the callback into an RPC callback request and at step 726 sends the RPC callback request, via plugin protocol session 415, to the stub plugin 324. At step 728 the plugin server 360 waits for an RPC response from the stub plugin 324, and upon receipt, ummarshals the RPC into parameters and generates a plugin API return to the endpoint plugin 370 at step 730, before returning to step 718 to wait for another event.
At step 732, the plugin server 360 receives an RPC specifying a window geometry change from the stub plugin 324. The RPC may include information about what sections of the plugin window 535 are overlapped or occluded, as well as any position or size changes of the actual plugin window 535 itself. In step 734 the plugin server 360 modifies the list of regions occluding endpoint-rendered user element 535 as needed to correspond with the received RPC, for example by informing the operating system 35 which areas of endpoint-rendered plugin window 535 should be rendered. In step 736 the plugin server 360 creates an RPC confirmation confirming that the window geometry change RPC was processed, and sends it to the stub plugin 324, via plugin protocol session 415, in step 738, before returning to step 718 to wait for another event.
At step 740, the plugin server 360 receives an RPC request from the stub plugin 324, and in step 742 unmarshals the RPC into parameters and generates a client plugin API call to the endpoint plugin 370 via client plugin API 365. At step 744 the plugin server 360 receives an API callback response, and at step 746 marshals the callback response into an RPC response, which the plugin server 360 then sends back to the stub plugin 324 in step 738 before returning to step 718 to wait for another event.
At step 748, the plugin server 360 receives an unload RPC request from the stub plugin 324, and in step 750 the plugin server 360 unloads the endpoint plugin 370. In step 752, the plugin server 360 sends an RPC confirmation confirming that the unload RPC was processed to the stub plugin 324, and then exits process 700 at step 754. At exit, the plugin protocol session 415 may or may not be terminated, depending on the particular embodiment. For example, if the user is still active, whether in the browser or another application, then the plugin protocol session 415 may be kept active until the browser is closed, the VDI session 405 is terminated, etc.
In sum, techniques are provided herein for establishing an integrated rendering of a browser window comprising user interface elements such as streaming media on a client endpoint device. A hosted web browser on an HVD draws a Hosted Virtual Desktop (HVD) image of a browser window and communicates it to the client endpoint device for display, the HVD display comprising at least one host-provided window element and at least one placeholder where client-provided data associated with a tag may be rendered. A client plugin server instantiates an endpoint browser plugin to render a tag in place of a portion of the HVD image before displaying an integrated display of the browser window and rendered tag content at the client endpoint device.
The above description is intended by way of example only. The description of has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more features, integers, steps, operations, elements, components, and/or groups thereof. The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed.
With respect to the Figures, which illustrate the architecture, functionality, and operation of possible implementations of methods, apparatuses, and computer readable media encoded with instructions, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometime be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
