Patent Application
20170010673
Computing technology has revolutionized the way we work, play, and communicate. Computing functional is obtained by a device or system executing software or firmware. Often, an important tool to allow users to influence the execution of software is via a user interface displayed on a display. The user interface may itself be the ultimate end point of the software.
In collaborative environments, user interfaces are often shared between users. Also, in remote access environments, a user interface of one display may be remotely accessed from another computing system. Often, it is the entire display that is shared. Examples of collaborative environments and technologies include electronic whiteboarding, collaborative authoring, tracking/revision marking, and so forth.
The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practiced.
At least some embodiments described herein relate to gesture recognition technology that allows a user to use gestures to share portions of a user interface (perhaps even by sharing the portions of the application that generate the user interface portion). A computing system, upon recognizing when a portion selection gesture has been entered on a display, associates the portion selection gesture with an associated portion of a user interface displayed on the display based on spatial relation of the portion selection gesture with the associated portion. In other words, the system estimates which user interface elements the user intended to select with the gesture. In response, the system causes the associated portion of the user interface to be shared for display on a remote display. In some embodiments, there may also be a target selection input received from the user that allows the system to identify which machines and/or users the selected user interface portion is to be shared with.
The portion selection gesture may be a positive gesture that is centered on the portion to be shared. Alternatively or in addition, the portion selection gesture may be a negative gesture (e.g., a redaction gesture) that centers over a portion of the user interface not to be shared. A compound selection gesture may include any number (zero or more) of positive gestures and any number (zero or more) of negative gestures to allow efficient entry of even complex selections of user interface elements.
By appropriate combination of position and/or negative gestures, fine-grained and efficient definition of the set of shared user interface element(s) may be made, and thus careful selection of shared user interface elements is enabled. This increases efficiency associated with sharing, and increases the user control over the sharing process.
This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description of various embodiments will be rendered by reference to the appended drawings. Understanding that these drawings depict only sample embodiments and are not therefore to be considered to be limiting of the scope of the invention, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
At least some embodiments described herein relate to gesture recognition technology that allows a user to use gestures to share portions of a user interface (perhaps even by sharing the portions of the application that generate the user interface portion). A computing system, upon recognizing when a portion selection gesture has been entered on a display, associates the portion selection gesture with an associated portion of a user interface displayed on the display based on spatial relation of the portion selection gesture with the associated portion. In other words, the system estimates which user interface elements the user intended to select with the gesture. In response, the system causes the associated portion of the user interface to be shared for display on a remote display. In some embodiments, there may also be a target selection input received from the user that allows the system to identify which machines and/or users the selected user interface portion is to be shared with.
The portion selection gesture may be a positive gesture that is centered on the portion to be shared. Alternatively or in addition, the portion selection gesture may be a negative gesture (e.g., a redaction gesture) that centers over a portion of the user interface not to be shared. A compound selection gesture may include any number (zero or more) of positive gestures and any number (zero or more) of negative gestures to allow efficient entry of even complex selections of user interface elements.
By appropriate combination of position and/or negative gestures, fine-grained and efficient definition of the set of shared user interface element(s) may be made, and thus careful selection of shared user interface elements is enabled. This increases efficiency associated with sharing, and increases the user control over the sharing process.
As the embodiments described herein may be implemented on a computing system, a computing system will first be described with respect to
Computing systems are now increasingly taking a wide variety of forms. Computing systems may, for example, be handheld devices, appliances, laptop computers, desktop computers, mainframes, distributed computing systems, or even devices that have not conventionally been considered a computing system. In this description and in the claims, the term “computing system” is defined broadly as including any device or system (or combination thereof) that includes at least one physical and tangible processor, and a physical and tangible memory capable of having thereon computer-executable instructions that may be executed by the processor. The memory may take any form and may depend on the nature and form of the computing system. A computing system may be distributed over a network environment and may include multiple constituent computing systems.
As illustrated in
In the description that follows, embodiments are described with reference to acts that are performed by one or more computing systems. If such acts are implemented in software, one or more processors of the associated computing system that performs the act direct the operation of the computing system in response to having executed computer-executable instructions. For example, such computer-executable instructions may be embodied on one or more computer-readable media that form a computer program product. An example of such an operation involves the manipulation of data. The computer-executable instructions (and the manipulated data) may be stored in the memory 104 of the computing system 100. Computing system 100 may also contain communication channels 108 that allow the computing system 100 to communicate with other message processors over, for example, network 110. The computing system 100 also includes a display 112 that may, for instance, display a user interface.
Embodiments described herein may comprise or utilize a special purpose or general purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are physical storage media. Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the invention can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.
Computer storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.
A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.
Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to computer storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile computer storage media at a computer system. Thus, it should be understood that computer storage media can be included in computer system components that also (or even primarily) utilize transmission media.
Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries or even instructions that undergo some translation (such as compilation) before direct execution by the processors, such as intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.
Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, pagers, routers, switches, and the like. The invention may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.
In this description and the following claims, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of “cloud computing” is not limited to any of the other numerous advantages that can be obtained from such a model when properly deployed.
For instance, cloud computing is currently employed in the marketplace so as to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. Furthermore, the shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly.
A cloud computing model can be composed of various characteristics such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud computing model may also come in the form of various service models such as, for example, Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). The cloud computing model may also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth. In this description and in the claims, a “cloud computing environment” is an environment in which cloud computing is employed.
The method 300 also includes recognizing when portion selection gestures have been entered on the display (act 302). For instance, in
In response, the portion selection gesture is associated with the selection of a corresponding portion of the user interface displayed on the display (act 303). For instance, the logic component 210 and/or the gesture recognition component 202 estimates a set of one or more user interface elements that the user intends to select based on the portion selection gesture. The portion selection gesture may be a positive (inclusion) gesture, in which case the selected portion is spatially related to the selected portion and is centered on the user interface element(s) to be selected. The portion selection gesture may alternatively be a negative (exclusion or redaction) gesture, in which case the selected portion is to be excluded from the selected portion.
The selected user interface portion is then shared (act 304) with another device. For instance, the control component 210 may cause the associated portion of the user interface to be made available to a remote display that is remote from the original display 112. In some cases, the sharing of the user interface portion (act 304) occurs not just by sharing the user interface portion itself, but by sharing a portion of the application that functions to generate that user interface portion with a remote computing system associated with the remote display. This will be described further below with respect to
In one embodiment, some user interface elements are shareable, and some are not. In that case, perhaps even if a user interface that is circled is not entirely sharable, the portion of the user interface that is circled and shareable may still be selected. For instance, if the entire user interface element 410 was not shareable, but the portion 415 was, then perhaps positive circling gesture 420 might cause only user interface element 415 to be selected.
The circling or encompassing gesture recognition may have a considerable degree of flexibility. As an example, the gesture might indicate that any user interface that is mostly (or with a certain percentage of area—such as 50 percent, 70 percent, 90 percent or the like) within the gesture is considered to have been selected. If the gesture does not represent a complete circling, then perhaps the two endpoints represented the incomplete ends are artificially joined in memory to determine whether enough of the user interface element is within bounds of the gesture to be considered selected.
Accordingly, as represented by
In one embodiment, rather that sharing just the user interface portion with the remote display, the portion of the application that generates the user interface portion is shared with the remote computing system associated with the remote display. The remote computing system may then run the application portion to result in the user interface portion appearing on its display.
As illustrated in
In the initial state 500A of
Now suppose that the application 500A is to be split. That is, suppose that the first user provides interaction or input suggesting that an application instance representing a portion of the larger application instance 500A is to be created for purposes of, at least temporarily, sharing the split application instance with a second machine and/or user. Such interaction might include the gestures described above. By so sharing, the associated user interface portion generated by the split application instances is also shared.
In any case, interaction and/or environmental event(s) are detected that are representative of splitting an instance of a smaller class off of the larger application class (act 601), thereby initiating the method 600 of
In the embodiment 700A of
In the embodiment 700B of
In the embodiment 700C of
In the embodiment 700D of
Note that there need be no change to the instance of the application 500 that is in state 500A from the perspective of the first machine and/or user. In that case, whatever endpoints are created for nodes 505 and 506 for the second machine and/or user may simply be cloned endpoints. During operation, if a cloned input endpoint received inconsistent input from both the first machine and/or user and the second machine and/or user, merging criteria may resolve the inconsistency. For instance, perhaps inconsistencies are resolved in favor of the delegating machine and/or user.
In an alternative embodiment, a remainder instance may be created that represents a logical remainder when the portion instance 500B is subtracted from the larger instance 500A, and thus no endpoint are cloned at all. For instance, in the case of
In operation, the first machine and/or user may maintain control or supervision over the actions of the second machine and/or user in interacting with the portion 500B of the application 500A. For instance, the second machine and/or user entity may be credentialed through the first machine and/or user with respect to the portion 500B such that data flows to and from the instance of the portion application 500B are approved by and/or channeled through the remainder of the application 500A controlled by the first machine and/or user. Furthermore, the access of the second machine and/or user to data (such as a data service) is strictly controlled. Data for nodes that are not within the portion application instances are provided via the approval of the first machine and/or user.
The larger application instance 801A is similar to the application 500A of
The portion application instance 801B is similar to the portion instance 500B of
The proxy service 810 provides a point of abstraction whereby the second machine and/or user 821B may not see or interact with the nodes 501 through 504 of the larger application instance 801A, nor may the second machine and/or user 821B interface with any of the endpoints of the nodes 505 and 506 that are assigned to the first machine and/or user 821A.
The proxy service 810 keeps track of which endpoints on node 505 are assigned to each node 505A and 505B, and which endpoints on node 506 are assigned to each node 506A and 506B. When the proxy service 810 receives input from the larger application instance (e.g., node 501), the proxy service 810 directs the processing to each of the nodes 505A and 505B as appropriate. Furthermore, when output are provided by the nodes 505A and 505B to the node 501, the proxy service 810 merges the outputs and provides the merged results to the node 501. For the perspective of the node 501, it is as though the node 501 is interacting with node 505, just as the node 501 did prior to application splitting. Accordingly, performance and function are preserved, while enabling secure application splitting, by maintaining appropriate information separation between the first and second machines and/or users 821A and 821B. Such merging of output results and splitting of inputs are performed by component 811 of the proxy service 810.
The proxy service 810 may also include a recording module 820 that evaluates inputs and outputs made to endpoints in each of the nodes 505A, 505B, 506A and 506B, and records such inputs and outputs. The recording module 812 also may record the information passed between nodes. Such recordings are made into a store 813. A replay module 813 allows the actions to be replayed. That may be particular useful if the portion application is assigned to another (i.e., a third) machine and/or user later on and a user of that third machine and/or user wants to see what was done. That third machine and/or user may come up to speed with what happened during the tenure of the second machine and/or user with the portion application.
The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
