This application relates in general to ubiquitous computing environment interfacing and, in particular, to a user interface and method for composing services in a ubiquitous computing environment through direction and selection operators.
Highly interconnected and widely distributed networking environments are becoming increasingly pervasive. Network infrastructures have become almost commonplace, with access points widely available in workplaces, homes, and public venues, such as libraries, malls, and coffeehouses. Network infrastructures are even available to travelers and commuters through wide area wireless services, such as WiMax and digital cellular networks.
A rich set of devices and services have now been made available by these distributed networking environments. Services include, for instance, data exchange, such as digital music download, interactive computer gaming, and conventional file sharing. To some extent, the devices available are driven by the services offered. For example, speakers or headphones are needed to listen to downloaded digital music and a gaming console may be required to participate in interactive computer gaming. Similarly, computer systems and application programs are needed access or store conventional files.
This combination of highly distributed and heterogeneous services and devices is known as “ubiquitous computing.” Activities in the ubiquitous computing paradigm center around service composition, which includes discovering services and devices available over the network, selecting a subset of desired services and devices, and connecting and controlling the selected subset. Accessing a single service is trivial, but access complexity rapidly increases as the range of combinable services and attendant devices also increase. The services can offer varied and widely diverging capabilities, and the number of services can potentially be quite large.
The richness of a ubiquitous computing environment can pose challenges to users in composing services. Ideally, the users should remain unaware of or be kept insulated from compatibility and interfacing issues at the device implementation level. Similarly, the users should not be mislead into assuming interconnectedness between disparate devices or services when a physical connection is not possible.
Locating suitable devices and services can be tedious and at odds with the goals of encapsulating implementation and physical interconnectivity details. Hosts, for instance, can offer assorted resources and capabilities through attached components. As needed, these resources and capabilities can be formed into a nested hierarchy that requires expanding and searching through successively detailed levels. Finding a device or service can potentially require extensive searching, which must be coupled with an ability to backtrack through the search space to resume searching at an earlier point. Nevertheless, the capabilities of available input devices vary widely and are typically limited to providing a minimal number of keys tied to performing a specific function. Ubiquitous computing environments often rely on mobile devices, such as cellular phones or “smart” watches, or simple end-user control devices, for instance, remote controllers for television or stereos. These devices have limited inputs, such as “jog” dials or a small set of keys that only allow direction and selection operations. Key sequences or combinations can sometimes be used to access further functions or extended character sets on these devices, but such solutions can be tedious and device-dependent.
Hierarchical menu navigational interfaces are widely used. A series of selections or parameters are provided in a tree-like structure, which is naturally intuitive and user friendly. Generally, hierarchical menus are offered for a specific device, such as a network service host that provides a library of media content. However, this model assumes that the device is tied to a particular service and the menu is limited to selecting a single “end point,” which is typically a target data source or destination. Support for interfacing to a device over a network infrastructure is not provided and the user must resort to manual means to locate, provision, and control distributed heterogeneous devices.
Therefore, there is a need for an approach to providing a flexible and simple user input interface for supporting service composition in a ubiquitous computing environment. Preferably, such an approach would support widely-distributed heterogeneous hosts, devices, and aggregates in a networked configuration using a minimal set of user operations.
One embodiment provides a user interface and method for composing services in a ubiquitous computing environment through direction and selection operators. A display of a ubiquitous computing environment is presented. One or more hosts operating in the ubiquitous computing environment is represented with host symbols. Components in the ubiquitous computing environment coupled to each of the hosts are represented with one or more component symbols per host symbol. Each such component provides a service or includes a device or aggregate component. The host symbols and the component symbols are navigated in response to user inputs. A direction operator provides a focus in the display on either one of the host symbols or one of the component symbols. A selection operator performed against either one of the host symbols or one of the component symbols triggers a service- or device-oriented task comprising one of discovery, selection, connection, and control.
A further embodiment provides a user interface and method for composing hierarchically-structured services in a ubiquitous computing environment through direction and selection operators. A graphical display of a ubiquitous computing environment is presented. One or more hosts operating in the ubiquitous computing environment are represented with graphical host symbols. Components in the ubiquitous computing environment coupled to each of the hosts are represented with one or more graphical component symbols connected to each graphical host symbol. Each such component provides a service or includes a device or aggregate component. Each such aggregate component includes a plurality of components. The graphical host symbols and the graphical component symbols are hierarchically navigated in response to user inputs. A direction operator provides a focus in the display on either one of the graphical host symbols or one of the graphical component symbols. A selection operator performed against either one of the graphical host symbols or one of the graphical component symbols triggers a service- or device-oriented task. The service- or device-oriented task is performed and includes one of discovery, selection, connection, and control.
Still other embodiments of the present invention will become readily apparent to those skilled in the art from the following detailed description, wherein are described embodiments by way of illustrating the best mode contemplated for carrying out the invention. As will be realized, the invention is capable of other and different embodiments and its several details are capable of modifications in various obvious respects, all without departing from the spirit and the scope of the present invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.
Ubiquitous Computing Environment
A ubiquitous computing environment is characterized by highly distributed and heterogeneous services and devices interconnected via a network infrastructure.
Each host is coupled to one or more components, which include aggregate and terminal components, and the host enables the components to be available over the network. Several components can be attached to a single host. An aggregate component is a combination of two or more components, which can also include nested aggregate components. For instance, the computer system 12 includes seven components, which include video camera 15, speaker 16, microphone 17, printer 18, file system 19, personal data assistant 20, and personal media player 21, which is an aggregate component. The personal media player 21 further includes individual media files 22a-d. Similarly, the computer system 13 includes a speaker 23, video display 24, and file system 25, which is also an aggregate component. The file system 25 further includes individual data files 26. Finally, the computer system 14 includes a file system 27 and video display 28. Other components are possible.
Each component, including aggregate components, can function as a data source or data sink, either separately or in combination. A data source provides data, whereas a data sink consumes data. For example, the video camera 15 and microphone 17 are data sources, whereas the speakers 16 and 23, video displays 24 and 28, and printer 18 are data sinks. The remaining components can function as both data sources and data sinks.
The hosts and components form a hierarchy of nodes within the environment. Each host forms the root of each node and the components form the individual leaves. Aggregate components form intermediate levels within the hierarchy, while non-aggregate components form terminal leaves. Other forms of host and component structuring are possible, although hierarchical structuring is intuitive and user friendly.
Services are provided through the components, including aggregate components. The components can also be treated as devices that perform a set of predefined services. For instance, playing music is a service, while a speaker is a particular type of device for playing music, among other functions. Generally, the devices available within a given environment 10 are driven to some extent by the services offered, although a richer set of devices could be provided, even if not all of the services utilize the available devices. Similarly, a richer set of services could be offered, even if not all of the devices support the available services.
In one embodiment, the hosts and components are interconnected using the Obje interoperability framework, which provides a set of scaleable and extensible interface elements that enable digital devices and services to interoperate over both wired and wireless networks, such as described in Black J. A. et al., “Speakeasy: A Platform for Interactive Public Displays,” ACM Conf. on Computer-Supported Cooperative Work (CSCW 2002), New Orleans, LA, Nov. 16, 2002; Edwards W. K. et al., “The Case for Recombinant Computing,” Xerox PARC Tech. Report CSL-01-1 (April 2001); and “Obje™ Interoperability Framework,” a white paper available at
http://www.parc.com/research/proiects/obje/Obie_Whitepaper.pdf, Xerox PARC (2003), the disclosures of which are incorporated by reference. Other forms of interconnection are possible.
Method
Service composition is provided in response to user interactions conveyed through a user interface.
Four basic tasks must be provided to support service composition, which include discovery 41, selection 42, connection 43, and control 44. In one embodiment, these tasks are performed through two user-performable operators: direction and selection. Discovery 41 allows a user to browse the lists of available hosts and each host's list of attached components, such as further described below with reference to
Selection 42 allows a user to designate a particular component to serve as a data source or data sink, such as further described below with reference to
Connection 43 provides the physical interconnectivity of the components at the selected end points, such as further described below with reference to
Finally, control 44 enables a user to monitor and sever active connections, such as further described below with reference to
State Diagrams
Service composition is provided through the user interface through a set of overlapping states that are navigated using the direction and selection operators. In one embodiment, the direction operator provides bidirectional rotation, although other forms of movement, including spherical, linear, and planar are possible. The selection operator provides a toggle movement, which can be of short or long duration, although other forms of movement are possible. Moreover, the direction and selection operators can be supplemented with various forms of feedback, including tactile and audible feedback. An example of a physical controller suitable for implementing the direction and selection operators is the PowerMate USB Multimedia Controller and Input Device, sold by Griffin Technology, Inc., Nashville, Tenn., which is an assignable and programmable controller that provides a rotary knob to enable both bi-directional rotary motion for direction and push-button motion for selection.
Service composition begins with discovery.
Within the set of components coupled to the host, the direction operator allows the user to shift the focus to each successive component (transition 57) and to return the focus back to the host (transition 58), thereby allowing the user to backtrack to resume discovery at an earlier point. The selection operator is bimodal. If the user performs the selection operator with a long duration movement while the focus is on an aggregate component (transition 59), the constituent components within the aggregate component are expanded within the display (state 53). If the component in the focus is itself an aggregate component (state 52), any previous levels within the hierarchy are automatically collapsed (transition 60) to limit the display to two levels of depth. Alternately, if the user performs the selection operator with a short duration movement (transition 61), service composition shifts to selection and connection (state 54), after which service composition returns to discovery (transition 62).
User Interface
The user interface for service composition can be implemented on a host, component, or other mechanism that is able to communicatively interface with the hosts and components in the ubiquitous computing environment 10.
The user interface 110 is loosely based on a “ball and stick” metaphor. Collections 111, 112, 113 of hosts and their coupled components are arranged with the host at the center and the components placed at equal distances around the host. Each host is represented by a host symbol 113, 116, 118. The components coupled to each host are represented by component symbols 115a-b, 117a-c, 119 respectively connected by lines to the host symbols 113, 116, 118. The focus is indicated by highlighting and an explanatory label in an explorer bar 120. For example, a focus on host symbol 114 corresponds to the explanatory label “Home Computer.” In a further embodiment, the host and component symbols could be replaced by icons, graphical symbols, shapes, or textual labels, as well as other forms, colors, and descriptions for the underlying host or component. Other user interface metaphors are possible.
Stacked Components
To maintain clarity of presentation, the number of component symbols allowed to be placed around a host symbol is limited.
Aggregate Components
An aggregate component is a combination of two or more components, which can also include nested aggregate components.
Referring next to
Component Selection, Connection, and Control
To insulate users from compatibility and interfacing issues at the device implementation level, the component symbols that represent incompatible and unconnectable components are logically “removed” from the set of component symbols displayed in the user interface following the selection of an initial component.
Visually, connection takes place in two stages.
While the invention has been particularly shown and described as referenced to the embodiments thereof, those skilled in the art will understand that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope.