Patent Application
20090273560
The present invention relates to computer user interfaces and, in particular, to a tangible user interface that is distributed and sensor-based.
Humans exhibit particular skill at sifting, sorting, and otherwise manipulating large numbers of small physical objects. When a human performs a task such as overturning a container of nuts and bolts and sifting through the resulting pile to find one of a particular size, or spreading photographs out on a tabletop and sorting them into piles, he or she uses both hands and all of his or her fingers actively and efficiently. However, when the human sorts digital information or media such as digital photographs or emails, the task typically does not effectively leverage the full range of physical manipulation skills. For example, one typical user interaction with a modern graphical user interface (GUI) is to click on an icon with a mouse, drag the icon to another location on the screen, and then drop it to reposition it or to assign the data it represents to a folder. This so-called “direct manipulation” of information, as afforded by a GUI, is a poor substitute for a human's facile all-finger, two-handed manipulation of physical items.
Tangible user interfaces (TUIs) have made some progress towards leveraging human physical manipulation abilities for interaction with digital information. For example, Fitzmaurice et al. pioneered physical ‘handles’ to digital objects, in which TUIs with handles operate by sensing the user's manipulation of each handle and displaying a co-located visual representation of the data being manipulated [Fitzmaurice, G. W., Ishii, H., Buxton, W.: Bricks, “Laying the foundations for graspable user interfaces”, Proceedings of CHI '05 (1995), 422-449]. Some TUIs, like the Designer's Outpost [Klemmer. S., Nevwnan, M., Farrell. R., Bilezikjian. M., Landay. J., “The designers' outpost: a tangible interface for collaborative web site”, Proceedings of UIST '01 (2001), 1-10] and DataTiles [Rekimoto, J., Ullmer, B., Oba, H., “Datatiles: a modular platform for mixed physical and graphical interactions”, Proceedings of CHI '01, New York, N.Y., USA, ACM Press (2001), 269-276], project graphics onto the handles themselves, while others, like Sensetable [Patten, J., Ishii, H., Hines, J., Pangaro, G., “Sensetable: a wireless object tracking platform for tangible user interfaces”, Proceedings of CHI '01 (2001), 253-360], utilize them simply as generic tangible cursors to the overlaid GUI. Advantages of TUIs over GUIs include support for two-handed input (though recent touch-screen interfaces also support this [see, e.g., Han, J. Y., “Low-cost multi-touch sensing through frustrated total internal reflection”, Proceedings of UIST '05, New York, N.Y., USA, ACM Press (2005), 115-118]), reduced cognitive load as compared to a GUI, faster target acquisition, greater facilitation of multi-person interaction, and a reduction in the level of indirection between a person's hand and the actual computation taking place when adjusting a parameter [Fitzmaurice, G., “Graspable User Interfaces”, PhD thesis, University of Toronto (1997)]. These features make handle-based TUIs more direct form of manipulation than a GUI alone.
Another class of tangible user interface largely dispenses with the GUI paradigm, featuring physical objects that directly embody the digital information or media that they represent. These TUIs do not implement handles to manipulate a GUI overlay; rather, such as in Ishii's Music Bottles [Ishii, H., Mazalek, A., Lee, J., “Bottles as a minimal interface to access digital information”, CHI '01, Extended abstracts on human factors in computing systems, New York, N.Y., USA, ACM Press (2001), 187-188] and Want et al.'s work with embedded RFID tags [Want, R., Fishkin, K. P., Gujar, A., Harrison, B. L., “Bridging physical and virtual worlds with electronic tags”, Proceedings of CHI '99, New York, N.Y., USA, ACM Press (1999), 370-377], the shape and features of the objects themselves suggest the semantics of the interaction. The inherent coupling between form and function in this class of TUIs brings an increased directness to an interaction with digital information or media. However, this gain in directness comes at a cost: since they represent the underlying data implicitly with their physical form, UIs featuring special-purpose objects can be more limited to a particular application domain or style of interaction.
Sensor networks consist of collections of sensing and communication devices that can be distributed spatially. They are capable of exhibiting coordinated behavior, forming a kind of “functional fabric” in the spaces that they inhabit without requiring external sensing or power infrastructure. Sensor network nodes can be built with an array of sensing technologies that can be used to build rich models of local interactions and their surroundings. The sensor network community has focused a great deal of effort on the technical issues that it faces. Many of these present themselves as tradeoffs in the design space. Longer battery life for a sensor node requires that other priorities, such as battery size, radio transmit range, or frequency of communication must suffer. The use of ultra-small components can cause the device to require more specialized and expensive assembly, so they may be avoided in favor of larger components, resulting in a larger device. Other challenges relate to implementing the desired sensor network behavior, such as coordinated sensing of events and mobile code that can be transmitted easily to another node and run immediately [Akyldiz, I F., Su, W., Sarkarasubramaniam, Y., Cayirci, E. “Wireless sensor networks: a survey”, Computer Networks, 38(4), (2002), 393-422].
While there has been a great deal of development activity directed to these and other aspects of wireless sensor networks in which many computationally-equipped nodes cooperate to perform a wide variety of tasks, a coherent set of design principles for Human-Computer Interaction (HCI) problems has not yet been developed. Little research effort has been invested in using sensor networks as user interfaces, and, in particular, in combining sensor network technologies with features from GUIs and TUIs. Such distributed TUIs (dTUIs), also called Sensor Network User Interfaces (SNUIs), in combination with a defined SNUI interaction language, could increase the directness of a user's interaction with digital information or media. Existing work relating to the concept of an SNUI interaction language by which a user might interact with digital information or media includes, for example, the “Smart-Its Friends” technique of pairing two personal devices by shaking them together at the same time is an example of grouping-by-gesture [Holmquist, L., Mattem, F., Schiele, B., Alahuhta. P., Beigi, M., Gellersen, H., “Smart-its friends: A technique for users to easily establish connections between smart artifacts”, Proceedings of UbiComp '01 (2001) 116-122]. In addition, Hinckley discusses several interactions based around the bumping of display screens into each other, including cooperative display sharing to create a larger viewing area for documents or photographs [Hinckley, K., “Synchronous gestures for multiple persons and computers”, Proceedings of UIST '03, New York, N.Y., USA, ACM Press (2003), 149-158]. Both projects make use of inertial data captured from an accelerometer, and are novel physical interactions with devices that could be adapted for use by an SNUI. BumpTop is a GUI-based interface that simulates the physics of real-world objects with inertia [Agarawala, A., Balakrishnan, R., “Keepin’ it real: pushing the desktop metaphor with physics, piles and the pen”, Proceedings of CHI '06, New York, N.Y., USA, ACM Press (2006), 1283-1292], and prototypes a wide range of gestural language primitives for interactions with icons.
The present invention is a distributed tangible user interface comprising compact tangible user interface (TUI) manipulative devices with sensing, display, and wireless communication capabilities and associated digital content management and other software applications. The manipulative devices can be physically manipulated as a group by a user in order to permit a user to efficiently interact with digital information, media, and interactive software programs. A group of manipulatives, used in concert, thus form a physical, distributed, gesture-sensitive, human-computer interface. In the preferred embodiment, each TUI manipulative has its own sensing, feedback, and communication abilities. A controller on each manipulative device receives and processes data from a movement sensor, initiating behavior on the manipulative and/or forwarding the results to a management application that uses the information to manage the digital content, software application, and/or the manipulative devices. TUI manipulatives according to the preferred embodiment can sense their neighbors, allowing management applications to utilize topological arrangement. They detect the proximity and identity of other manipulative devices, responding to and/or forwarding that information to the management application, and may have feedback devices for presenting responsive information to the user. They can also be used to implement gestural interaction languages and HCI applications.
In one aspect, a tangible user interface according to the invention comprises a plurality of tangible user interface manipulative devices, each tangible user interface manipulative device being independently manipulable relative to the other tangible user interface manipulative devices. Each tangible user interface manipulative device comprises at least one wireless communications device, a visual display for digital content, a power source, at least one movement sensor, and at least one controller. The controller receives data from the movement sensor, processes the received data to derive movement parameters, and possibly forwards the derived movement parameters or initiating tangible user interface behaviour in response to the derived movement parameters. An associated management application sends digital content or behavior instructions to individual tangible user interface manipulative devices, receives derived movement parameters from at least one of the tangible user interface manipulative devices, processes derived movement parameters to derive instructions about management of the digital content or program behaviour, and changes program behavior or manages the digital content according to the derived instructions. The management application may send at least one of revised digital content or behavior instructions to individual tangible user interface manipulative devices according to the derived instructions. The tangible user interface manipulative devices may also comprise at least one neighborhood wireless communications device for sensing nearby tangible user interface manipulative devices, with the controller being further adapted for sensing the position of at least one nearby tangible user interface manipulative device, processing the position of, and any communication received from, the sensed nearby tangible user interface manipulative device in order to derive neighborhood information, and possibly forwarding the derived neighbourhood information to the management application or initiating tangible user interface behavior in response to the derived neighbourhood information.
In another aspect, the present invention is a method for facilitating user interaction with digital content or application programs, comprising the steps of displaying a visual representation of at least one of digital content or program control elements on a plurality of tangible user interface manipulative devices, such that a subset of the digital content or program control elements is displayed on any individual device, detecting at least one of a manipulation of at least one of the tangible user interface manipulative devices or a location-based relationship between at least two of the tangible user interface manipulative devices, and deriving digital content relationship information or instructions from the detected manipulation or relationship.
Other aspects, advantages and novel features of the invention will become more apparent from the following detailed description of the invention when considered in conjunction with the accompanying drawings wherein:
The present invention is a platform that applies technology and methodology from wireless sensor networks to tangible user interfaces in order to yield new possibilities for human-computer interaction. The distributed tangible user interface, or Sensor Network User Interface (SNUI), of the present invention is a platform for physical interaction with information and media, comprising compact devices with sensing, display, and wireless communication capabilities. These devices can be physically manipulated as a group in order to permit a user to efficiently interact with digital information and media. The present invention permits people to interact with information and media in physical, natural ways that approach their interactions with physical objects in their everyday lives.
As used herein, the following terms expressly include, but are not to be limited to:
“Manipulative”, “Tangible user interface manipulative”, “Tangible user interface manipulative device”, and “TUI manipulative” all mean compact devices with sensing, display, and wireless communication capabilities that can be physically manipulated as a group in order to permit a user to interact with digital information and media in physical manner.
“Sensor network” means a wireless sensor network in which each node is physically independent, self-powered, and can communicate with its peers via a wireless radio.
“Sensor Network User Interface” and “SNUI” mean a system comprising a set of tangible user interface manipulatives that simultaneously provide the functionality of both a sensor network and a user interface. Although in “Experiences and Directions in pushpin computing” [Lifton, J., Broxton, M., Paradiso, J., Proceedings of the 4th international symposium on Information processing in sensor networks (2005)], Lifton et al. use the acronym SNUI to describe a separate interface allowing a user to interact with a sensor network, the term as used herein refers to a sensor network and user interface that are effectively congruent.
“Siftables” means a specific preferred embodiment of a tangible user interface manipulative, a set of such manipulatives, or a system employing the same.
The Sensor Network User Interface of the present invention is a distributed TUI in which a set of small physical manipulatives have sensing, wireless communication, and user-directed output capabilities. These devices can host a representation of a wide range of data and control parameters and can be physically manipulated as a group as a tangible interface to the data and applications employing the control parameters. They provide a generic interaction platform that combines the flexible graphical display capabilities of a GUI, the physicality of a TUI, and the capabilities of a sensor network. In contrast to TUIs that provide handles to a projected digital representation of data, an SNUI operator holds a representation of the data or parameter itself that can be perceived and altered directly. Though it increases system complexity for designers, the greater functional capabilities at the level of the individual interaction node minimize or eliminate requirements for external infrastructure such as a projector or an augmented tabletop surface. Also, an SNUI according to the present invention can be more easily reconfigured than the corresponding manipulatives of a single-purpose TUI built to utilize a particular sensing modality and form factor.
The component TUI manipulative devices are compact electronic devices with motion sensing, display, and wireless communication capabilities. One or more of such devices may be physically manipulated by a person in order to interact with digital information, media, and applications. A group of manipulatives can thus act in concert in order to form a physical, distributed, gesture-sensitive, human-computer interface. In the preferred embodiment, each TUI object is stand-alone (battery-powered and wireless) and does not require installed infrastructure such as specialized sensing surfaces, large displays, instrumented tables, or cameras in order to be used. Each manipulative has its own sensing, feedback, and communication abilities, effectively making a set of manipulatives a combination of a tangible interface and a sensor network. TUI manipulatives according to the preferred embodiment can sense their neighbors, allowing applications to utilize topological arrangement. They can also be used to implement any number of gestural interaction languages and HCI applications.
Advantages and features of the preferred embodiment include, but are not limited to, provision for physical interaction with a collection of wirelessly-networked devices, the ability to manipulate data in 3-dimensions rather than being restricted to a 2-dimensional surface such as a screen, enablement of multi-person interaction, and provision for on-device input and output. Unlike other tabletop systems having collections of ‘pucks’ that interact with a larger display surface, the present invention provides input (motion sensing) and output (a graphical display) on the manipulative itself. The manipulatives are self-contained and mobile, being lightweight, portable, and battery-powered and they do not require any external sensing infrastructure, such as sensing tables or cameras).
In one preferred embodiment, called “Siftables”, the TUI manipulatives are independent, compact devices with sensing, graphical display, and wireless communication capabilities.
Graphical display 205, such as, but not limited to, a color screen, allows the TUI manipulative to display graphical content. The screen may be redrawn at high enough frame rates to create moving images or animations, or it may be updated more slowly. In the siftables implementation, processing system 210 controls screen 205, and it may provide it with vector drawing commands, bitmaps for display, or both. Processor 210 can also put display 205 into a low-power sleep state, can adjust the brightness and contrast, and can control a number of other parameters. Processing system 210 may be implemented in any of the many ways known in the art, including, but not limited to, as a single multiprocessor or as a multi-processor system. In the present siftables design, processing system 210 comprises a main processor for handling most tasks and a secondary processor for handling infrared communication with, and detection of, neighboring siftables.
In the siftable implementation, each siftable has flash memory module 225 that is separate from processing system 210. This memory can be written and read by processor 210, initiated either directly by a program running on the processor, or as a result of communication over the radio from a remote software program. Arbitrary data may be stored in this memory, such as images for display on the screen, variable names and associated values, samples from the accelerometer, program code, or any other data that an application may require. The microprocessor may also retrieve a sequence of images stored in this memory and display them sequentially on the screen, creating visual animations or movies.
The tangible user interface manipulative of the present invention is preferably powered on by the user. This is accomplished using button 240 in the siftables implementation, but it will be clear to one of skill in the art that any of the many means of powering on a small handheld device known in the art would be suitable including, but not limited to, causing power on to occur in response to a pre-specified gestural maniplations (such as, but not limited to, turning the device upside down for more than a prespecified time duration or shaking it) or in response to a wireless command from another device including, but not limited to, a computing device or another tangible user interface manipulative.
The tangible user interface manipulative is preferably powered by battery 255, but it will be clear to one of skill in the art that many other means of powering a small hand-held device are suitable for use in the invention including, but not limited to, the use of an on-board solar cell and associated charging system Battery 255 is preferably rechargeable, but clearly may be any type of battery known in the art that can power the electronics and will fit into the allotted space. The “siftables” implementation is driven by small lithium-polymer batteries that may be recharged. The rechargeable battery can be recharged via charging port 250, which is a micro-USB socket attached to the main circuit board. A micro-USB cable is inserted into this socket, and the other end is inserted into a specialized charging printed circuit board that was designed for this purpose. It will be clear to one of skill in the art that alternatives, such as, but not limited to, inductive charging, may be advantageously employed to improve this characteristic so that frequent recharging is not required.
Sensing in the “siftables” implementation is accomplished using accelerometer 220 and four IrDA transceivers 230. When manipulated atop a flat surface, a siftable can sense its own motion in the plane of the surface, as well as the impacts it has with other objects. It can also sense the action of being lifted, tilted, or shaken, or vibrations resulting from the surface itself being impacted, such as, for instance, if the user raps their knuckles against the surface. The four IrDA transceivers are tuned for extremely short-range communication, on the order of 1 cm, and are used to detect neighboring manipulatives at close range. The sensed information can then be used by the siftable itself to modify its own behavior, or it can be shared wirelessly with other siftables or with a nearby computer. These sensing, graphical display, and wireless communication capabilities permit a set of siftables to behave as a single, coordinated interface to information and media.
It will be clear to one of skill in the art of the invention that the system details of the “siftables” implementation represent only one exemplary implementation of the present invention and that many alternate configurations are also within the scope of the present invention. For example, the particular sensing and wireless communication details, as well as the architecture of the software, could be implemented differently in order to achieve the same capabilities. For instance, infrared communication may be replaced with capacitive data transmission, or Bluetooth radios replaced with Zigbee compatible radios. However, despite any such variations in implementation details, the user experience and application possibilities remain similar. It is also possible to implement a richer set of input and output capabilities that can be used to implement new application scenarios, such as additional sensing modalities, such as, but not limited to, capacitive, and output capabilities, such as, but not limited to, auditory or tactile. On-siftable feedback may be provided by the graphical display, or by vibrational/haptic actuation, emission of sound, or other output capabilities that may be added to the siftable by an optional connected circuit.
For a typical data manipulation task, each TUI manipulative is populated, for example, via radio, with a representation of a single instance of the data to be manipulated, or with a control parameter than can be adjusted. The user's physical manipulations to the collection of manipulative devices are sensed and used as input to the system. Visual feedback during the task is presented to the user on the LCD display, and auditory feedback can be played by a nearby computer. In preferred embodiments, the TUI manipulatives have the ability to sense their arrangement with respect to each other using infrared communication capabilities, the ability to sense their own motion or the motion of the surface or object that they are in physical contact with using their accelerometer, and the ability to wirelessly communicate with each other, with a nearby computer, with a mobile phone, and/or with another electronic device.
The run-time behavior of a manipulative may be determined by a program that is installed directly on the manipulative, by software running remotely on another computer that communicates with the manipulative wirelessly over the radio, or both. These models for application development represent distinct options for developers, and a given application may rely on a combination of local and remote code. Remote software may issue commands that alter the running behavior of the manipulative. Remote software can also ‘subscribe’ to receive periodically collected data (streams) or to detected changes in inertial or neighborhood state (events). It may also command the manipulative to start or stop broadcasting its ID over infrared, to start or stop listening for neighboring manipulatives, to display shapes, arbitrary graphics, or images retrieved from memory on the screen, to power off, or to take a number of other actions. Remote software may also be used to recognize gestures occurring across more than one manipulative, and may issue commands to the manipulative(s) in response to these gestures. The manipulative may generate a response to each command received from remote software, and may modify its internal state and current behavior accordingly. A software library may encapsulate the text-based language protocol, allowing for behavior specification to take the form of function calls in an application programming interface (API). The utility of such an API is that it creates the software abstraction of a software object by which a single manipulative may be controlled. Thus, monitoring and controlling a group of physical manipulative devices is accomplished in a manner very similar to monitoring and controlling a group of software objects, which enables straightforward application development.
Inertial Data Handling. On a given time interval (100 Hz in the current system), the current values from a three-axis accelerometer are read, and processed. The results may be handled by code on the manipulative or may be optionally reported over the radio. If remote software has ‘subscribed’ to receive raw accelerometer data, the raw values are transmitted over the radio. Using the newly captured values and a buffer of previous values, values such as running mean and variance are computed for each axis. If remote software has ‘subscribed’ to either of these values, the newly-computed values are transmitted over the radio. A current estimate of tilt and shaking state are computed. The current tilt and shake values are compared to the previously-measured values from the last analysis cycle, and if the current values are different from the previous ones and remote software has ‘subscribed’ to events for either of these values, the current state is transmitted over the radio.
In the present “siftables” implementation, a secondary processor is responsible for transmitting and receiving messages to and from nearby siftables using infrared communication. This duty may alternately be performed directly by the main processing unit, or may be performed by the main processing unit in conjunction with dedicated extra hardware. A nearby siftable that is close enough to be in infrared communication is considered a ‘neighbor’, and neighbors can be sensed in the four horizontal (North/South/East/West) directions surrounding a siftable. Transmitting and listening behavior may be turned on or off by the main processor. If a new neighbor arrives on a side, the stored representation of the current neighborhood is updated to reflect this addition and the updated state is immediately communicated to the main processor. The frequency of infrared messaging attempts is preferably high enough so that both arrivals and departures seem immediate to the user.
To transmit, the siftable periodically ‘pings’ an infrared pulse in each direction, and if a reply ‘ping’ from a neighboring siftable is received, it transmits a message to the neighbor, communicating the siftable's ID and from which side the message emanated. In order to reduce ‘jitter’ in the form of spurious arrival or departure messages to the main processor due to infrared message collisions or intermittent failures in infrared communication, a neighbor must not be heard from for a given amount of time, or a given number of communication cycles, before it is considered to have departed and its departure is noted by the processing unit. By this policy, new neighbor arrivals are communicated immediately, and departures take slightly longer to be confirmed and communicated. It will be clear to one of skill in the art that other modes of short-range data transmission may be used in place of infrared, and other communication schemes could be used in place of the particular “handshaking” and jitter-reduction algorithm used currently.
Neighbor communication behaviour may be changed as the program runs, including actions such as, but not limited to, enabling or disabling infrared listening or broadcast behaviour, updating broadcast information such as the siftable's numerical ID or broadcast period, querying information from a secondary processing unit, and/or commanding a secondary processing unit to perform some other duty that it is capable of, such as, but not limited to, user-directed feedback.
In preferred embodiments, the tangible user interface manipulative devices of a set have the ability to recognize and interact directly with each other.
In some embodiments, derived movement parameters may be used only by the manipulatives themselves, rather than, or in addition to, being sent to a management application, or such parameters may not be utilized at all. In one exemplary embodiment of such a system, code programmed into the microcontroller handles the inputs such as sensing of neighbors and accelerometer data, and generates appropriate output, such as, for example, animating through sequences of images that are stored in the flash memory. This code utilizes the same basic subroutines that might alternatively be implemented by a remote program over the radio link, but in this embodiment the manipulative never forwards any data, nor receives any commands, from a remote machine.
In one application that has been prototyped using the “siftables” implementation, there is no radio communication at all. Each siftable has a portrait of a person on it, and when two siftables are placed next to each other, the portraits animate to look towards each other. If more than one neighbor is detected at a time, the portrait animates to look around in a confused way. When they are moved away, the portraits animate to looking back forward again. If a siftable is shaken, the portrait animates to look around in a confused way, and if it is tilted, the portrait looks down, in the direction of gravity. Both neighbor- and motion-detection are used only on-board the siftables.
Rudimentary communication with the exemplary “siftables” implementation employs ASCII characters that may be typed on a keyboard. This language is not as efficient as it could be if binary opcodes and values were employed, but it makes it easier to interact with a siftable directly if needed, by typing characters by hand over a serial port type connection and viewing the immediate results. The decision to favor transparency over efficiency in the “siftables” implementation was made deliberately, in order to make debugging and programming for the “siftables” implementation easier, but it will be clear to one of ordinary skill in the art that other communication methodologies may be advantageously, and likely more efficiently, employed in the present invention. The Microsoft Windows-compatable RealTerm terminal program may be advantageously used to interact directly with the siftables, and any programming language with serial port or Bluetooth capabilities such as the Python programming language may be used to script application behavior for siftables.
The serial port parameters for connecting to a siftable over Bluetooth are: 115200,8,N, 1. To initiate communication with a siftable of the current implementation, after opening the Bluetooth serial port, the message “1234\n” is sent to the siftable. If the siftable receives this message successfully, it replies with “ok 1234\r\n”. The general format of communication with a siftable is to send an ASCII command, terminated by an endline ‘\n’ character. The siftable replies to every message that it receives, and its reply is always terminated by “\r\n”. If the command is a query, and the query does not generate a valid reply, or is mis-typed, for instance, if “var get dave\n” is sent to a siftable and there is no variable named “dave”, the siftable will reply with an error message, such as: “error, no variable named dave\r\n”. If the siftable cannot parse the command that it was sent, for instance if a command is sent that doesn't make sense, like: “app delete at slot monkey\n”, the siftable will similarly reply with an error message. If the message sent to a siftable is not a query, for example if “acc smooth on\n” is sent to initiate low-pass filtering on the accelerometer data, the siftable will reply with “ok acc smooth on\r\n”.
Table 2 lists the ASCII Language commands that may be sent to a current-version siftable over the radio, and the reply that can be expected in response. This is not a final version of the language specification, but is rather a snapshot of the language used for the present “siftables” embodiment. It will be clear to one of skill in the art that commands may be added, removed, or modified with respect to this version.
A software library that encapsulates the text-based language protocol for the “siftables” implementation, allowing for behavior specification to take the form of function calls in an application programming interface (API), has been implemented in the Python programming language, and features a nearly one-to-one correspondence between the existing language commands and the corresponding functions that it provides. Table 3 presents the Siftable Python API Listing. It will be clear to one of skill in the art that this program listing is exemplary, and that many other similar protocols, languages, etc might be advantageously employed in an implementation of the present invention.
One useful aspect of the present invention is that it provides a platform upon which an interaction language for SNUIs can be developed. The interactions that comprise such a language are physical manipulations to single or multiple TUI manipulatives that can be sensed with the onboard sensors. A library of manipulations and metaphors, analogous to point-and-click or drag-and-drop for the GUI but related specifically to the SNUI, can be developed, with customization possible for each SNUI and/or application. In certain applications, the system can optionally permit user customization of the gestural library.
It will be clear to one of skill in the art that the present invention may be advantageously employed in a wide rage of applications, including, but not limited to, media manipulation and management (such as photo sorting), use as a live performance instrument for video and/or audio, editing of video and/or audio, project planning, meeting management, game platform, educational activities, picture-in-picture system/TV control, ambient physical information widgets, user interface device for Computer-Aided-Drafting (CAD), wearable social network display, and financial monitoring and manipulation. In any of these applications, it is clear that the ability of the invention to synchronize actions performed using the manipulatives with a representation of the same data on a computer or an Internet-based system provides valuable functionality. While not all applications will require this synchronization between the manipulatives and a computer or an Internet-based system, it is an option that may be advantageously provided for any of them.
Media organization and manipulation system. TUI manipulatives according to the present invention can visually represent media, such as, but not limited to, digital song files, photos, videos, and emails, by displaying a small visual representation of the media on their screens. Other manipulatives may optionally represent tags, labels, or titles for collecting and grouping of content. Using the TUI manipulatives, the media can be organized by physical manipulation, and these manipulations may in turn create effects on the computer where the original content resides. For instance, a tag manipulative may be brought near an existing group in order to attach a label to the content or to organize those media in an album on the user's computer. Media and tags can be changed or manipulated via individual gestures.
A task particularly well-suited to the present invention is digital photograph organization. For example, a group of photos from a user's camera might include a series of images from the user's latest vacation. Thumbnails of the photographs to be sorted are transmitted wirelessly to the TUI manipulatives by a host computer. The user physically creates groupings by pushing the manipulatives into piles. The devices sense these movements and impacts using their accelerometers, and use their radios to share information about these events amongst each other. When more than one manipulative is bumped at nearly the same time, a grouping is created back on the host computer. The photographs on the user's computer are then automatically placed into a folder together. Bumping a “label” manipulation with the word “vacation” into the group can then apply the label to the group, naming the group on the user's computer and grouping the images for the user's convenience in later browsing. It can be seen that, using the present invention, the task of sorting digital images is now much closer to a physical photograph organization activity that leverages a users' manual dexterity.
Live performance instrument. In one embodiment of this application, TUI manipulatives are used as an instrument or interface for the live production of audio or video streams. Each manipulative corresponds to audio and/or video clips, to a “sequence/measure” container, to live video and/or audio streams, or to visual and/or audio effects. The user can arrange the sequence manipulatives in a row to form a timeline. The timeline itself can be represented visually as a line running across all the manipulatives that are arranged edge to edge, indicating that they are part of a temporal progression. A visual cursor sweeps across this line, showing the playback position of the media. The media is played through external (off-manipulative) speakers or displays in real time, controlled at least in part by information transmitted wirelessly via the manipulatives' radios. Effects can be applied by either touching an “effect manipulative” to a clip or by gesturing with the “clip manipulative” itself For example, a ‘reverb’ effect manipulative may be applied to a manipulative representing an audio sample such as a guitar riff. The sample immediately acquires a reverb sound wherever it appears in the currently active sequence, and aspects of this sound may be manipulated by movements to the manipulative itself or to the “reverb” manipulative. Some ‘global’ manipulatives may also affect the entire stream at once. For example, a tempo manipulative can be tilted back and forth to affect the overall speed at which the cursor sweeps through the timeline. All of these manipulations control a sequence in real-time, which may be simultaneously presented on a large display or displays and/or a set of audio speakers.
Editor for video/audio. Like the live performance instrument, in this application manipulatives represent clips or effects. But instead of controlling a display or speaker in real-time, this tool allows the user to construct a sequence of video clips using manipulatives in order to edit together a final piece. Clips can be edited on-manipulatives (e.g., rolls, ripples, trims, and other manipulations) using gestures, and the result may be previewed on the manipulative's screen. These same gestures apply to the live performance instrument as well. The user can arrange the manipulative clips linearly or in a branching manner to explore and edit timelines quickly, trying different possibilities by re-arranging the relevant manipulatives. These timelines represent edits of the designated clips and can be viewed on a large display. Again, ‘global’ manipulatives may affect the entire timeline; for example, one manipulative may allow the user to scrub through the edited timeline shown on the large display.
Project planning. This application is a physical and interactive way to produce flow charts and ‘Gantt’-style charts. TUI manipulatives may represent people, actions or states of a project or process, and they may be arranged into a diagram to create orderings and dependencies. Real-time feedback (visual on the manipulatives, visual on a nearby screen or projection, auditory, or tactile on the manipulatives themselves) may notify the user of problems or other relevant status. The structure of the process model or chart is constructed as the individual manipulatives are placed proximately to each other and they wirelessly communicate their neighbor state to a nearby computer where the overall structure is kept up-to-date with the real-world manipulations. In this way the manipulatives provide a real-time constraint satisfaction solver, featuring physical manipulation elements, but in which the resulting structure is captured digitally and saved for future examination, manipulation, and distribution. The application can feature two-way updates between the physical representation on the manipulatives and a software application.
Meeting Management. To schedule and organize meetings, TUI manipulatives may represent people or organizations. The user may arrange manipulatives into groups to schedule a meeting with the people represented. Conflicts may be transmitted wirelessly by radio from the user's computer (which has access to the schedules of the other participants) and indicated visually to the user in real-time via the graphical display. This application is an example of a ‘constraint satisfaction’ application, in which the user is attempting to organize data, and wherein some organizations of data produce conflicts, such as scheduling a meeting with a worker at a time in which he or she is not available. In this, and other constraint satisfaction problems, the manipulatives help the user to quickly experiment with different arrangements to find a valid solution without creating conflicts. At a meeting, people and action items can each be represented by individual manipulatives that show a visual representation of their designation. To accept a task, a meeting participant bumps their manipulative into a task manipulative. Visual feedback on the individual participant manipulatives may illustrate which tasks the participant has agreed to, as well as other features of the commitment such as estimated time or cost, other resources required, etc. During the meeting, physical manipulations of the TUI manipulatives may permit their use as a voting interface or as a way to subtly annotate or comment on the ongoing meeting in real-time. Interactions during the meeting can be captured and wirelessly transmitted by radio to a nearby computer, where they can be saved to calendars and other productivity software, and communicated to the participants for later review.
Game platform. There are a large number of games, such as, but not limited to, yu-gi-oh, magic: the gathering, mahjong, scrabble, boggle, and dominos, that currently make use of non-electronic tokens or objects. TUI manipulatives can be used as active tokens for these games, augmenting the games with responses to physical gestures and spatial configurations, automatic state-tracking and scorekeeping, visual feedback and engaging animations. For instance, in an augmented game of dominoes, each manipulative can display the visual representation of a single domino and uses its infrared neighbor-detection to determine when it is placed next to another manipulative. Visual feedback can be generated to indicate whether a given arrangement is valid with respect to the rules of the particular game. The manipulatives can also show visual feedback indicating to which player they belong, with, for example, a uniquely colored border or by a characteristic icon displayed in some location on the screen. Physical manipulations of the manipulatives during these games, such as arranging them spatially, or moving them gesturally, can contribute new elements to the existing structure of the game. For instance, in an augmented version of Magic: The Gathering, spells may be cast by moving a manipulative in a spatial pattern (1-dimensional, 2-dimensional, or 3-dimensional), and battles may be fought by placing a manipulative from one player's collection next to a manipulative from the other player's collection. The manipulatives can show a visual representation of the character or the function that they represent at each moment, and as the game progresses story-enhancing animations could be shown on the manipulatives. Details of the spatial arrangement could have meaning to the gameplay as well. For example, placing two manipulatives face-to-face could initiate a battle, while placing them side-by-side could initiate a cooperative action in the game.
Educational activities (language, vocabulary, math, logic, etc.). In an educational setting, TUI manipulatives may be used to implement learning activities, wherein the manipulatives display visual representations of content, and learners place the devices into spatial arrangements that reflect their understanding of the content. For instance, manipulatives could visually display symbols such as letters, numbers, variables, operators, chemical elements, and the learner could arrange them into linear sequences or two-dimensional topologies in order to form sentences, equations, molecules, and more. The end results may correspond to ‘correct’ or ‘wrong’ arrangements based on the task domain and the spatial configuration that the user created, and visual feedback can indicate this. Alternately, the visual content displayed on the manipulatives' screens can change such that a valid answer is displayed whenever the user places the manipulatives into a configuration. The manipulatives sense their neighbors using infrared, and the overall topology is determined either by the manipulatives themselves or by software running on the server with which the manipulatives are in wireless communication. For each arrangement, or at other moments during the interaction, the system can compute the overall or partial arrangements and present the learner with immediate feedback about their arrangement via on-manipulative or auditory feedback. This system could be viewed as a constraint-satisfaction application, and this type of visual representation and on-manipulative feedback is applicable to a wide range of similar applications. Alternatively, the system can log all of the student's arrangements as part of a more creative exercise, such as narrative creation, and the results can be visualized on the learner's computer or on the internet during the process or afterwards.
Picture-in-picture system/TV control. TUI manipulatives can also be used in conjunction with larger screens. For instance, used with a television screen, a manipulative can be used to implement a feature like “picture-in-picture”, showing a continually-updated visual feed from a channel different from the channel being shown on the main display. This may be accomplished by wirelessly transmitting a live video stream, or by sending periodically updated still images from the monitored secondary channel to the manipulative. The origin of this stream or procession of images can be either from the television itself if it features wireless communication, or from a computer working in conjunction with the television, such as a home media center or a “set-top box”. The communication can optionally be bi-directional as well; gestural interaction (for example: lifting, shaking, tilting) may, for example, be used to change the channel on the television to the channel being monitored on the manipulative. An extension of this configuration would be that a collection of multiple manipulatives can each show previews of separate channels, and physically manipulating a given manipulative in a particular manner (for example: lifting, shaking, tilting) could switch the television to show that particular channel. Other gestures might reveal the upcoming schedule of the represented channel; for example, tilting the manipulative left to right would scroll through that timeline. Shaking the manipulative might tell a DVR or set-top box to record the selected program. These interactions would rely on wireless communication between the television/computer/set-top-box and the manipulative or collection of manipulatives.
Ambient physical information widgets. In this application, TUI manipulatives display live information feeds, such as, but not limited to, weather, news headlines, or stock information. This information is transmitted wirelessly to the manipulative from a nearby server computer connected to the Internet. The visual rendering may be done on the manipulative, or may be computed on the server and transmitted as complete images to the manipulative. Each manipulative could alternately show a video feed, or some other visual representation of activity at a remote location. For instance, the activity at a motion sensor in an elderly relative's house could be shown. The sensor and data collection system at the remote location is connected to the Internet and periodically uploads its current state. Then, the Internet-connected server at the user's location retrieves this data and makes it available to the manipulative (or the manipulative might directly access this information itself, depending on its communication capabilities). The manipulatives can be arranged in a user's physical space, for instance on their (physical) desktop, bedside table, kitchen counter, or in some other location. Each manipulative is thus a part of a user's physical life/work space and shows continually updated information in a manner that can be viewed and understood quickly with little attention focus required.
UI device for Computer-Aided-Drafting (CAD). A single TUI manipulative, or group of manipulatives can be used as a control interface for computer-aided-drafting software, replacing or supplementing the computer mouse. Lifting and moving the manipulative can fluidly change the angle of view on a larger computer screen. The manipulatives' screens may show which tool they represent or which visual projection they are designated to manipulate. This is an example of a general class of UI possibilities in which a set of manipulatives replaces or supplements the existing mouse and keyboard, offering concurrent multi-point discrete or continuous control into a software application on a computer. Additionally, in this application or in others, TUI manipulatives may be used as a “window” into particular parts of a computer-generated virtual space; for instance, showing a view of the three-dimensional space that updates as the user moves the manipulative around physically.
Wearable Social Network Display. A visual representation of a user's social network identity can be displayed on a TUI manipulative that they carry with them or that they wear as a piece of jewelry on their body, clothing, or personal possessions, such as backpack or shoulder bag. The manipulative can wirelessly retrieve updated information about the user's profile, and optionally the profiles of the user's contacts as well, from the user's personal computer when the user is at home, or can access this information when the user is away from the computer or otherwise “on the go” by connecting to the user's mobile phone. At any time, the manipulative can be used both as a display, showing elements from the user's online profile, and as an interface to manipulate the profile, allowing the user to modify the profile and forge new connections with other users in the physical world. The manipulative may have access to all of the user's online information, or to only a subset of this information; for example, it may be able to display the user's profile picture and to transmit the user's profile URL, email address, or other information to another user's manipulative.
With one such an application, when users are in the same physical place, they may use their manipulative together to access content from each others' profiles or to manipulate their profiles in real-time. For instance, if two users place their manipulatives next to each other, this expression of intimacy could create a ‘friend’ relationship in their social network representation or could strengthen an already-existing connection. These real-world interactions and updates may be exposed to contacts in a user's social network in the form of a “feed” or other information representation. An example of this is the “feed” mechanism in Facebook, where the online interactions that users or their contacts engage in are made visible to other participants as a continually updated log. The user's profile may also be edited using the manipulative, by using gestures such as tilting, shaking, or 3D spatial movements in order to select information for inclusion or exclusion from on-the-go interactions. These manipulations of the users' social network representation may propagate immediately to change the online representation, if the manipulative can access the Internet via its radio either directly or through a nearby computer or mobile phone, or they may be stored for update at a later time when such network access becomes available.
Financial monitoring and manipulation. Similar to the Ambient Physical Information Widgets application, in this application each TUI manipulative shows an information feed—in this case related to financial information. The content for this information feed may be collected from online sources by software running on a server (either in the user's location, or remotely), then transmitted wirelessly to the manipulative. Software on the manipulative shows the information in an appealing and easily glance-able manner, in order to allow a user to efficiently monitor a number of separate information feeds. For instance, a manipulative might show the current price of a stock, the difference in value from a previous reading, or a longer-term graphical summary. The difference between this application and the Ambient Physical Information Widgets application is that here the manipulative can also be used as an interface for navigating the information and for making transactions. For instance, tapping on a manipulative might change the visualization currently being displayed. By shaking or tilting a manipulative, the owner's holdings in a particular stock could be increased or decreased immediately. These interactions rely on a wireless connection to software on a server, which would have access and authority to make transactions with the user's accounts. This connectivity permits a collection of TUI manipulatives to become an active part of a trader's information-rich environment that may currently be dominated by large passive display screens.
In another possible application for financial purposes, a certain group of TUI represent investment options—for instance, various stocks, mutual funds, or certificates of deposit. Each manipulative displays which option it represents. One manipulative is the “action” manipulative, and shows a distinct visual image indicating it as such. Placing the “action” manipulative next to an investment manipulative initiates investment from the user's financial account into that particular option, either all at once, or in such a manner that the amount invested depends on a continuous parameter such as tilt, or the length of time that the manipulatives are kept proximate. Visual feedback on the manipulatives indicates the success of, or the degree of, the transaction. The transactions can be made immediately, or the record of the interactions can be kept, and a “commit” action at the end of the interaction (either using a manipulative or using a computer) can trigger the action to be taken. The manipulatives have wireless communication with a server, which has network-based access to financial accounts and the ability to make transactions on the user's behalf.
In another possible financial application, a group of TUI manipulatives represents a user's accounts and the investment or money-management options offered by a financial institution. A user may be at home, or they may be at the location of the institution in consultation with a member of the institution. The manipulatives display a visual representation of the account, instrument, or action that they represent, and financial arrangements such as the purchase or adoption of certain financial instruments (stocks, bonds, etc.) or the transfer of money between accounts, can be achieved by manipulation of the manipulatives representing these entities. Again, the transactions may be made at the time of the interaction or later, and the manipulative have wireless communication with a server that has network-based access to financial accounts and the ability to make transactions on the user's behalf.
The present invention takes design principles for addressing human-computer interaction problems and applies sensor network technologies to them in order to both yield new kinds of tangible interfaces and new design principles specific to the possibilities inherent in Sensor Network User Interfaces. The tangible user interface manipulatives of the present invention give direct physical embodiment to information items and digital media content, allowing people to use their hands and bodies to manipulate these data instead of relying on virtual cursors and windows. By leveraging people's ability to manipulate physical objects, the present invention radically simplifies the way people interact with information and media and enables a new degree of directness in physically manipulating and interpreting information and media.
While a preferred embodiment is disclosed, many other implementations will occur to one of ordinary skill in the art and are all within the scope of the invention. Each of the various embodiments described above may be combined with other described embodiments in order to provide multiple features. Furthermore, while the foregoing describes a number of separate embodiments of the apparatus and method of the present invention, what has been described herein is merely illustrative of the application of the principles of the present invention. Other arrangements, methods, modifications, and substitutions by one of ordinary skill in the art are therefore also considered to be within the scope of the present invention, which is not to be limited except by the claims that follow.
This application claims the benefit of U.S. Provisional Application Ser. No. 61/063,479, filed Feb. 4, 2008, the entire disclosure of which is herein incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61063479 | Feb 2008 | US |
