Patent Grant
8352853
The present invention relates in general to the creation and conveyance of multi-model content. More particularly, the present invention relates to the encoding of device independent multi-model content for conveyance between potentially disparate devices.
There is a trend toward creating more immersive experiences, which in effect often entail creating experiences which involve a larger number or more of one's senses. At least a couple of examples include surround sound, wide screen displays, and the incorporation of vibratory effects. However in order to convey a more immersive experience, the experience will often involve a more complex set of signals having a larger number of components, which in turn require an increasing amount of network bandwidth and device output resolution, when the set of signals for creating the experience are conveyed between devices.
Enhancements in at least some experiences, such as multimedia experiences and/or gaming are increasingly including greater complexities in audio, video and haptics, which are principally experienced as part of previously authored or one-way content for rendering or play back on a target device, or where the interaction within the framework of the content presentation is largely scripted. However, as of yet, adoption of corresponding effects in two-way, i.e. user-to-user, communications has been extremely slow and/or largely non-existent. This may be attributable at least in part to the differences in the rendering capabilities of different end user devices and lack of a framework within which increasingly multi-modal experiences can be readily created and composed.
The present inventors have recognized that the creation and encoding of multi-modal content can be further facilitated by establishing a vocabulary of standard predefined effects in the form of tokens, which can be used to minimize the amount of information needed in creating, and subsequently communicating an effect, which can then be mapped, and scaled if necessary, to effect producing elements in a particular device.
The present invention provides for a method for encoding device independent multi-modal content including an arrangement comprising one or more of a plurality of tokens, each token corresponding to a standard predefined effect. The method includes defining a token for one or more standard predefined effects, wherein the token is adapted for being mapped to one or more effect producing elements in a target device. One or more tokens are then composed for playback together as an arrangement.
In at least one embodiment, the method further comprises associating a gesture with each of the tokens, receiving one or more gestures, and determining the token associated with each of the received gestures.
In at least a further embodiment, the method further comprises communicating the arrangement to a target device, mapping the tokens to one or more effect producing elements in the target device, and activating the one or more effect producing elements for which a token has been mapped in the target device in accordance with the arrangement of tokens.
The present invention further provides for a composer circuit for encoding device independent multi-modal content. The composer circuit includes a user input adapted for receiving a user selection corresponding to one or more tokens associated with one or more standard predefined effects. The composer circuit further includes a controller adapted for forming an arrangement of one or more tokens intended to define the one or more standard predefined effects to be produced by one or more effect producing elements in a target device.
The present invention still further provides for an effect producing circuit for playback of an arrangement of one or more tokens. The effect producing circuit includes a receiver for receiving an arrangement of one or more tokens associated with one or more standard predefined effects, a storage element including one or more prestored standard predefined effects, each prestored standard predefined effects being associated with a corresponding one of the one or more tokens. The effect producing circuit further including one or more effect producing elements, and a controller adapted for mapping the standard predefined effects associated with the tokens in the received arrangement with the one or more effect producing elements.
These and other features, and advantages of this invention are evident from the following description of one or more preferred embodiments of this invention, with reference to the accompanying drawings.
The present invention will be described by way of exemplary embodiments, but not limitations, illustrated in the accompanying drawings in which like references denote similar elements, and in which:
As required, detailed embodiments of the present invention are disclosed herein, however, it is to be understood that the disclosed embodiments are merely exemplary of the invention, which can be embodied in various forms. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely serve as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present invention in virtually any appropriately detailed structure. Further, the terms and phrases used herein are not intended to be limiting, but rather, to provide an understandable description of the invention.
The wireless devices 12 in some instances may be self contained and operate independently. In other instances, the wireless communication device will couple to a terminal device 16, and will facilitate the terminal device communicating with the network 14. At least some wireless communication devices 12 will couple to the network 14 via a base station 20, or other type of wireless transceiver, which is physically coupled to the network 14 via a wired connection.
The network 14 may include any type of network that is capable of sending and receiving communication signals. In at least one embodiment, the network 14 includes a wireless communication network. The network 14 may also include or be in communication with a data network, such as the Internet, an Intranet, a local area network (LAN), a wide area network (WAN), a cable network, and other like communication systems. The network 14 may also include or be in communication with a telecommunications network, such as a local telephone network, long distance telephone network, cellular telephone network, satellite communications network, cable television network and other like communications systems. Furthermore, the network 14 may include or be in communication with more than one network and may include a plurality of different types of networks. Thus, the network 14 may include or be in communication with a plurality of data networks, a plurality of telecommunications networks, a combination of data and telecommunications networks and other like communication systems.
The exemplary communication network can additionally include one or more devices 18, which are coupled to the network via a wired connection, like a personal computer. For example, the personal computer can be coupled to the network via a dial-up modem or a cable modem connection. Additionally coupled to the network are various servers, which supply various types of information, such as web pages, internet address to domain name translations, and the like. In some instances, the servers supply user accessible content, which is of direct interest to the user. In other instances, the servers supply information useful in managing the operation of the network.
In at least one embodiment of the present invention, a multi-modal content effects server 22 is provided. In the illustrated embodiment the server 22 is coupled to the network via a wired connection, however, it is also possible for the server to support a wireless connection to the network. The multi-modal content effects server 22 could also be integrated as part of the network 10. In some instances, the functionality of the server can be incorporated in one or more other network elements, including a target device, which is intended to receive the arrangement of tokens for playback, and a device upon which the multi-model content is encoded.
The composer circuit 30 includes a user input 32 adapted for receiving a user selection corresponding to one or more tokens associated with one or more standard predefined effects, which can be reproduced on a target device. The standard predefined effects are generic descriptions, which can be mapped onto one or more of the effect producing element associated with a corresponding target device for producing an intended effect. Generally, one or more discernable inputs are associated with one or more corresponding tokens, such that when a discernable input is detected, the corresponding token is added to an arrangement of tokens being composed.
In some instances, the tokens may relate to generic envelope definitions of a signal to be conveyed to at least one of the one or more effect producing elements. The user input can include one or more elements for receiving input from a user. Examples of elements forming all or parts of the user input 32 include a proximity sensitive surface 34, such as a touch sensitive screen, one or more user actuated switches 36, such as those contained as part of a keypad, pressure sensor(s) 38, microphone(s) 40, and/or camera(s) 42.
A proximity sensitive surface 34, such as a touch screen, which is capable of detecting movement across the surface in at least a two dimensional plane, could be used to detect one or more gestures that are each associated with one or more corresponding tokens. A gesture can correspond to the tracing as defined by the end of a stylist or the end of a user's finger, as the outline of the gesture is traced along the proximity sensitive surface 34. Other forms of user input can be used to alternatively identify and select desired tokens, and/or be used to select and/or identify one or more modifying parameters, and their corresponding values.
The composer circuit 30 additionally includes a controller 44, which is adapted to form an arrangement of tokens based upon the tokens selected by the user and correspondingly identified by the circuit 30. In at least some embodiments, the arrangement of tokens is produced as part of an encoding module 46, which forms part of the controller 44. In at least some instances, the controller incorporates one or more microprocessors, which execute routines written in executable code, as well as other circuitry for managing the control of the operation of the composer circuit 30 including the encoding module 46.
The encoding module 46, as well as any other modules which are included as part of the controller 44, can incorporate routines written in executable code and/or other circuitry for managing the functionality of the composer circuit 30 including forming an arrangement of one or more tokens. In some instances any executable code in the form of one or more sets of prestored instructions 48 could be stored in a storage element 50, which may be coupled to the controller and included as part of the composer circuit 30. Alternatively, any prestored instructions could be included as part of the controller 44. The prestored instructions could take the form of firmware, microcode, or a file containing a list of executable instructions. The storage element 50 could additionally contain one or more prestored gestures, tokens and/or effects 52, and any association therebetween, which might be used by the controller 44 in facilitating the forming of an arrangement.
By using tokens corresponding to standard predefined effects, an identification of a token in an arrangement can be used to reference an effect, such as a waveform profile and/or an envelope definition of a signal, which might otherwise require a larger amount of information to be represented adequately. In at least some embodiment, a standardized set of envelope definitions are known to the target device, which can be appropriately referenced by identifying the corresponding token identifier. In addition to envelope definitions, some of the tokens can be used to reference execution modifiers, such as a token which directs the envelope definition associated with another token to be repeated multiple times, or a token which marks the end of an arrangement. It is further possible to modify and/or further define a token using one or more parameters having a suitable value. In this way an amount of customization and/or modification can take place relative to a generic envelope definition without requiring that an entire waveform be reproduced and communicated as part of an arrangement. In this way a more bandwidth efficient encoding of multi-modal effects can be performed.
The effect producing circuit 60 includes a receiver 62 for receiving an arrangement of one or more tokens. In at least some embodiments, the arrangement comprising one or more of a plurality of tokens, is received as part of at least one of a message, and a file, such as an SMS message, or a MPEG or MIDI file. The arrangement could also be potentially sent as part of an active circuit switched or packet data telephone call connection.
In at least some embodiments, the receiver will be coupled to an antenna 64 for facilitating wireless communications between the source of the arrangement and the effect producing circuit 60. In other instances the receiver 62 may be more directly coupled, via a tethered connection, to the source of the arrangement.
The effect producing circuit 60 additionally includes a storage element 66, which includes one or more prestored standard predefined effects 68, which are each associated with a corresponding token. The effect producing circuit 60 further includes one or more effect producing elements 70, which are capable of rendering at least one type of effect. At least a couple of examples of effects include lighting effects, nonverbal sound effects, and vibrational effects. Examples of effect producing elements include light emitting elements 72 (i.e. Light Emitting Diodes (LED's)), displays 74 (i.e. Liquid Crystal Displays (LCD's)), speakers 76, transducers 78, and rotary mass vibrators 80. In some instances some effect producing elements may be capable of producing more than one type of effect. For example, a multifunctional transducer can produce both an audio and a vibrational output. One skilled in the art will realize that other types of devices, capable of producing the types of effects specifically noted above, as well as other user perceivable effects, are possible without departing from the teachings of the present application.
The effect producing circuit 60 still further includes a controller 82, which has a mapping module 84 that is adapted for mapping the standard predefined effects associated with the tokens in the received arrangement with the one or more effect producing elements. In some instances, a token will be directed to a type of effect that is not supported by the target device and/or is not perceivable by the intended user. In these instances the affected effects may be either ignored or remapped to another type of effect. Such a remapping may be effective in accommodating users having a particular disability. For example, the hearing impaired may elect to map audio effects to either a vibrational effect producing element or a lighting effect producing element. Such a setting could be stored as a user preference, or could make use of a default setting based upon a known condition.
In mapping an effect to one or more particular effect producing elements, an effect may have associated location characteristics, which might dictate the particular effect producing element for use, which is preferred. For example, a device may have multiple audio producing elements, which produce a signal which are discernable as originating from different locations, such as from the right or from the left. In these instances, a particular parameter which defines source location may be used. Still further it may be possible to send suitable signals to more than one audio producing element to create effects having still greater variety and/or granularity in terms of their perceived characteristics. In some instances it may be possible to capture location based information relative to the effect, when such information for an effect can be detected and correspondingly encoded.
Still further, the playback of the effect in the target device can additionally accommodate differences in rendering capabilities by scaling the effects before the effects are forwarded to the corresponding effect producing elements. Such a scaling could be performed locally in the controller 82 as part of a scaling module 86.
In at least some instances, the controller 82 can include a timing module, which allows multiple tokens to be rendered during overlapping time segments, where at least a portion of one effect is rendered during the same time period as the rendering of at least a portion of another effect associated with another token. The overlap during playback of at least a pair of effects can be accommodated through the use of temporal parameters, which can be used to more directly define the time during which the effect is rendered by the corresponding effect producing elements 70. Such effects could be managed with the assistance of a timing module 88, which in at least some embodiments is part of the controller 82.
Similar to the encoding module 46 in conjunction with the composer circuit 30, the mapping module 84, the scaling module 86, and the timing module 88 can include executable code in the form of one or more sets of prestored instructions 90, which can be stored in the storage element, for execution by the controller 82. Similar to the controller 44 of the encoding circuit 30, the controller 82 of the effect producing circuit 60 in at least some embodiments can incorporate one or more microprocessors, which execute routines written in executable code, as well as incorporate other circuitry for managing the control of the operation of the effect producing circuit 60.
By controlling the rendering on a target device, wherein an effect producing circuit 60 has information concerning the rendering capabilities of the target device, and the relationship between the target device and the standard predefined effects is similarly known, any conversion of the signals to be forwarded to the one or more effect producing elements 70 which best represents the effect on the target device can be determined, thereby enabling the multi-modal content to be more device independent.
While the effect producing circuit 60 described in connection with
In at least some embodiments, the storage element may additionally include prestored standard predefined effects 106, and one or more sets of prestored instructions 108 in support of the operation of the one or more controller modules.
The network interface 96 facilitates communications between the server and the network 14 including any devices coupled to the network 14, but which is not otherwise coupled to the server 22.
In the illustrated embodiment, at least two tokens corresponding to an execution modifier are illustrated. A first execution modifier has a gesture 120 represented by all four sides of a square, beginning in the lower left corner, moving up, to the right, moving down, and to the left. This gesture 120 represents a repeat, which can be used to modify the number of times the previous token is repeated. The last token has a gesture 122, which includes a left to right horizontal line, which then doubles back, partially, angling downward. This gesture 122 represents a send command marking the end of the current arrangement.
While each token generally has a standard predefined effect, certain characteristics can be modified by identifying one or more parameters, and corresponding values for each of the parameters. In some instances, a parameter can be applied to the entire token. In other instances, a parameter can be applied to modify a portion of the standard predefined effect. Some parameters allow for different values to be applied to specific time slices.
In some devices, where the multi-modal effect is being composed, the device may have a numeric keypad, which facilitates easier entry of a numeric value. In devices that do not include a numeric keypad alternative ways to enter a number may be possible. For example, a number could be entered using a gesture on a proximity or touch sensitive surface. Alternatively, a key could be depressed a number of times corresponding to the desired number to be entered. One skilled in the art will readily appreciate that there are numerous ways in which different types of data such as numbers could be entered without departing from the teachings of the present invention.
In some instances, the type of effect may similarly need to be defined, as the same generic envelope definitions may be applicable to more than one type of effect. Where a token has been associated with a lighting effect, at least one parameter may enable the composer to include a parameter defining a color value for all or a part of the generic envelope definition.
Parameters affecting start times and end times can similarly be used, which will allow tokens to be defined for different effects, which are defined to occur during at least partially overlapping or simultaneous time segments, which would allow the possibility of some effects being time synchronized.
The middle effect consisting of four repeated bump pulses, corresponding to each of effect producing elements “A” and “B”, and which are slightly offset, could be the result of a single effect having five repeated bump pulses, which has been defined as having a location characteristic that started on the left and moved to the right. In a device having vibrational effect producing elements, which are respectively located in each of the four corners, the “A” line effect might be sent to both the upper and lower left vibrational effect producing elements, and the “B” line effect might be sent to both the upper and lower right vibrational effect producing elements. Presumably, the effect producing circuit 60, would know the device specific characteristics, and would appropriately map the effects to the effect producing elements 70.
In devices which have only a signal vibrational effect producing element, the “A” line effect could be merged with the “B” line effect, which may be a desirable matching for that particular device. In this way, the intended effects can be appropriately mapped to a particular device pursuant to the capabilities of the device.
If a global lighting feature does not exist for the device, the “C” line effect may be remapped to flash any existing LCD. If the device has no lighting effects capability, or if the user is visually impaired and has set user preferences to remap lighting effects, the “C” line effect can be either ignored or remapped to a sound effect producing element or a vibrational effect producing element per user preference, or default setting (i.e. in absence of a user preference).
It is further possible that supplemental sensing and effect producing capabilities could be added to a particular device in the form of an accessory, which is coupled to the target device. In such a case, the effect producing circuit would need to establish access to the supplemental effect producing elements, and the module performing the mapping 84 or 100, would need to be aware of the extended capabilities. Similarly, the composer circuit would need access to the accessory to be able to interact with the additional capabilities. Such an accessory could include a cover and/or bag which extends around the original device or alternatively could be an accessory which resides proximate to the device, like a sense pillow, shaped similar to a mouse pad, with which the user can alternatively interact.
While the preferred and other embodiments of the invention have been illustrated and described, it will be clear that the invention is not so limited. For example, while several different tag embodiments have been illustrated and described, one skilled in the art will readily appreciate that still further tag embodiments are possible without departing from the teachings of the present invention. Numerous modifications, changes, variations, substitutions, and equivalents will occur to those of ordinary skill in the art without departing from the spirit and scope of the present invention as defined by the following claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4698672 | Chen et al. | Oct 1987 | A |
| 4980844 | Demjanenko et al. | Dec 1990 | A |
| 5359712 | Cohen et al. | Oct 1994 | A |
| 6463440 | Hind et al. | Oct 2002 | B1 |
| 6801604 | Maes et al. | Oct 2004 | B2 |
| 6816737 | Kim | Nov 2004 | B2 |
| 20020194388 | Boloker et al. | Dec 2002 | A1 |
| 20030184498 | Blumberg et al. | Oct 2003 | A1 |
| 20040066932 | Seligmann | Apr 2004 | A1 |
| 20040111673 | Bowman et al. | Jun 2004 | A1 |
| 20040207542 | Chang et al. | Oct 2004 | A1 |
| 20050010638 | Richardson et al. | Jan 2005 | A1 |
| 20060085767 | Hinckley et al. | Apr 2006 | A1 |
| 20060232614 | Leroux | Oct 2006 | A1 |
| 20070282797 | Wang et al. | Dec 2007 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20070005790 A1 | Jan 2007 | US |
