Patent Application
20200076866
Embodiments hereof relate to systems, devices and methods for streaming haptic effects. In particular, embodiments hereof provide techniques for authoring, encoding, streaming, decoding, and playback of haptic effects from a haptic effects author to one or many recipient client haptic devices.
Haptic effects, alone or in combination with traditional media formats, including audiovisual formats, provide an enhanced user experience. Haptic effect streams can be combined with audiovisual streams to enhance the user experience with streaming media, whether it be live streaming media or pre-recorded streaming media. Client haptic devices include one or more haptic actuators to deliver the haptic effects of a haptic effect stream to a user. Different models of client haptic devices, however, may include different sets of actuators and therefore different output capabilities. Output capabilities may differ in terms of the haptic effects that are available, the magnitude of output, the bandwidth of output, and other characteristics of haptic effects.
Difference in output capabilities of various models of client haptic devices can lead to difficulties in designing haptic effects. For example, an operator or user preparing haptic effects, e.g., a haptic designer, may have to decide whether to design haptic effects including only the subset of effects that a majority of devices can reproduce or to take advantage of new and differing haptic capabilities. Even common actuators for common haptic effects may differ in individual specifications and characteristics. Designing different haptics streams for multiple different devices is impractical and burdensome.
These and other drawbacks exist with conventional haptically enabled display devices. These drawbacks are address by the inventions described herein.
Embodiments of the invention include an end-to-end system for one-to-many haptics stream authoring, streaming, and playback. A haptic authoring system receives a stream of haptic effects, either pre-recorded or authored by an operator as they are sent to the system. The stream of haptic effects may also include a “haptic track,” i.e., a series of haptic effects. The authoring system breaks down each of the haptic effects in the stream into haptic blocks of content identifiers by encoding the haptic effects according to a library of associations between basic haptic elements intended by the operator and content identifiers. Instead of transmitting the raw data of the haptics stream, the authoring system transmits the haptic blocks to a haptic server system. The haptic server system connects with haptic client devices, supplies them with device-specific libraries for decoding the haptic blocks, and transmits the haptic blocks to the haptic client devices. The haptic client devices then decode the haptic blocks by associating the received content identifiers with basic haptic elements playable by the haptic client device. Accordingly, the haptic designer creates a single haptics stream which may then be encoded and transmitted to multiple haptic client devices of differing capabilities for playback.
In an embodiment, a computer implemented method of encoding haptic inputs for transmission is provided. The method is performed on a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method. The method comprises accessing a master haptics library storing associations between content identifiers and designed haptic effects, the designed haptic effects including a first haptic block template and a second haptic block template. The method further comprises receiving haptic effect information; generating a first haptic block according to the first haptic block template based on the haptic effect information; and attaching a first timestamp to the first haptic block according to the haptic effect information. The method further comprises generating, from the master haptics library, a second haptic block according to the second haptic block template based on the haptic effect information; attaching a second timestamp to the second haptic block according to the haptic effect information; and encapsulating the first haptic block and the first timestamp with the second haptic block and the second timestamp into a haptic data unit. The method further comprises transmitting the haptic data unit to a haptic server for distribution to one or more client haptic devices.
In an embodiment, a system for encoding and transmitting a stream of haptic effects is provided. The system comprises one or more processors and a memory storing instructions. When executed by the one or more processors, the instructions cause the system to access a master haptics library storing a plurality of haptic block templates, each haptic block template representing a haptic element and having block characteristics, and to receive haptic effect information. The instructions further cause the system to generate, from the master haptics library, a first haptic block having a first block template and first block parameters according to the haptic effect information, attach a first timestamp to the first haptic block according to the haptic effect information, and to generate, from the master haptics library, a second haptic block having a second block template and second block parameters according to the haptic effect information. The instructions further cause the system to attach a second timestamp to the second haptic block according to the haptic effect information, encapsulate the first haptic block and the first timestamp with the second haptic block and the second timestamp into a haptic data unit, and transmit the haptic data unit to a haptic server for distribution to one or more client haptic devices.
In an embodiment, a computer implemented method of transmitting a stream of haptic effects is provided. The method is performed on a haptic server having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method. The method comprises establishing communication with a haptic authoring system; receiving from the haptic authoring system a master haptics sub-library comprising a subset of a master haptics library stored on the haptic authoring system; and establishing communication with a client haptic device. The method further comprises accessing a device haptics library; providing to the client haptic device a device specific haptics sub-library from the device haptics library according to a device identification of the client haptic device and the master haptics sub-library; and receiving a master haptics stream from the haptic authoring system; generating a client haptics stream from the master haptics stream. The method further comprises transmitting the client haptics stream to the client haptic device.
In an embodiment, a haptic server system for transmitting a stream of haptic effects is provided. The haptic server system comprises one or more processors; and a memory storing instructions. When executed by the one or more processors, the instructions cause the system to establish communication with a haptic authoring system, receive from the haptic authoring system a master haptics sub-library comprising a subset of a master haptics library stored on the haptic authoring system. The instructions further cause the system to establish communication with a client haptic device and access a device haptics library. The instructions further cause the system to provide to the client haptic device a device specific haptics sub-library from the device haptics library determined according to a device identification of the client haptic device and the master haptic effects sub-library, receive a master haptics stream from the haptic authoring system, generate a client haptics stream, and transmit the client haptics stream to the client haptic device.
In an embodiment, a computer implemented method of decoding a client haptics stream for playback is provided. The method is performed on a client haptic device having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the client haptic device to perform the method. The method comprises establishing communication with a haptic server; obtaining a client haptics library from the haptic server, wherein the client haptics library is generated by the haptic server according to a device identification of the client haptic device. The method further comprises receiving, by the client haptic device, a client haptics stream from the haptic server, the client haptics stream comprising a plurality of haptic blocks and a corresponding plurality of timestamps. The method further comprises generating a plurality of haptic effects for playback on one or more actuators of the client haptic device, each haptic effect selected from the client haptics library according to a content identifier of a corresponding haptic block from the client haptics stream to be played back at a time in accordance with a timestamp of the corresponding haptic block; and outputting the plurality of haptic effects to the one or more actuators of the client haptic device.
In an embodiment a client haptic device for decoding a client haptics stream for playback is provided. The client haptic device comprises one or more processors; and a memory storing instructions. When executed by the one or more processors, the instructions cause the client haptic device to establish communication with a haptic server, obtain a client haptic effects library from the haptic server, wherein the client haptic effects library is generated by the haptic server according to a device identification of the client haptic device, receive the client haptics stream from the haptic server, the client haptics stream comprising a plurality haptic blocks and a corresponding plurality of timestamps. The instructions further cause the client to generate a plurality of haptic effects for playback on an actuator of the client haptic device, each haptic effect selected from the client haptics library according to a corresponding haptic block from the client haptics stream to be played back at a time in accordance with a timestamp of the corresponding haptic block and output the plurality of haptic effects to the actuator of the client haptic device.
The foregoing and other features and advantages of the invention will be apparent from the following description of embodiments hereof as illustrated in the accompanying drawings. The accompanying drawings, which are incorporated herein and form a part of the specification, further serve to explain the principles of the invention and to enable a person skilled in the pertinent art to make and use the invention. The drawings are not to scale.
Specific embodiments of the present invention are now described with reference to the figures. The following detailed description is merely exemplary in nature and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background, brief summary or the following detailed description.
Embodiments of the present invention are directed to the creation, transmission, and playback of data streams of encoded haptic effects, referred to herein as haptics streams. A haptic authoring system is provided to receive haptic effects, encode the haptic effects according to a master haptics library, and transmit the haptic effects as a master haptics stream. A haptic server system is provided to receive the master haptics stream from the haptic authoring system and parse the master haptics stream into client haptics streams for client haptic devices. The haptic server system also provides and updates client haptics libraries permitting each specific client haptic device to decode the client haptics streams for playback on the specific device. The haptic server system may also be configured to provide additional media content to client haptic devices for playback, including audiovisual streams, for example.
The haptic authoring system 100 may be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device, and/or other device that can be programmed to receive and encode haptic effects. In an embodiment, any or all of the functionality of the computing haptic authoring system 100 may be performed as part of a cloud computing platform.
The haptic authoring system 100 includes one or more processors 110 (also interchangeably referred to herein as processors 110, processor(s) 110, or processor 110 for convenience), one or more storage device(s) 120, one or more haptic input device(s) 160, and/or other components. The processor 110 is programmed by one or more computer program instructions stored on the storage device 120. For example, the processor 110 is programmed by an authoring user interface system 151, a haptic encoding system 153, an encapsulating system 154, and an authoring communications system 155. Additionally, the storage device 120 stores a master haptics library 190. As used herein, for convenience, the various instruction modules and systems will be described as performing an operation, when, in fact, the modules and systems program the processor 110 (and therefore the haptic authoring system 100) to perform the operation. Further details and features of the haptic authoring system 100 configured for implementing features of the described invention may be understood with respect to
Storage devices, as discussed herein, include non-transitory computer readable media, and to any media or computer memory that stores data and/or instructions that cause a machine to operate in a specific fashion. Such non-transitory media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, as well as various forms of solid state media. Volatile media includes dynamic memory. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, DVD, Blu-Ray, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, and networked versions of the same.
The haptic server system 200 may be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device, and/or other device that can be programmed to receive and transmit haptics streams. In an embodiment, any or all of the functionality of the haptic server system 200 may be performed as part of a cloud computing platform.
The haptic server system 200 includes one or more processors 210 (also interchangeably referred to herein as processors 210, processor(s) 210, or processor 210 for convenience), one or more storage device(s) 220, and/or other components. The processor 210 is programmed by one or more computer program instructions stored on the storage device 220. For example, the processor 210 is programmed by a server communications system 255, a client stream creation system 253, and a client identification system 251. Additionally, the storage device 220 stores a device haptics library 290. As used herein, for convenience, the various instruction modules and systems will be described as performing an operation, when, in fact, the modules and systems program the processor 210 (and therefore the haptic server system 200) to perform the operation. Further details and features of the haptic server system 200 configured for implementing features of the described invention are described with respect to
The client haptic device 300 may be configured as a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device, a smartwatch, video game controller or console, virtual reality (VR), augmented reality (AR), or mixed reality (MR) headset, and/or other device that can be programmed to receive haptics streams and output haptic effects. In an embodiment, any of the computing functionality of the haptic client device 300 may be performed as part of a cloud computing platform.
The client haptic device 300 includes one or more processors 310 (also interchangeably referred to herein as processors 310, processor(s) 310, or processor 310 for convenience), one or more storage device(s) 320, and/or other components. The processor 310 is programmed by one or more computer program instructions stored on the storage device 320. For example, the processor 310 is programmed by a playback user interface system 351, a haptic playback system 352, a haptic decoding system 353, a decapsulating system 354, and a client communications system 355. Additionally, the storage device 320 stores a client haptics library 390. As used herein, for convenience, the various instruction modules and systems will be described as performing an operation, when, in fact, the modules and systems program the processor 310 (and therefore the client haptic device 300) to perform the operation. The client haptic device 300 also includes one or more haptic output devices 330. Further details and features of the client haptic device 300 configured for implementing features of the described invention may be understood with respect to
The haptic output devices 330 may be or include any suitable output device known in the art. For example, the haptic output devices 330 may include thin film actuators, such as macro-fiber composite (MFC) actuators, piezoelectric material actuators, smart material actuators, electro-polymer actuators, and others. The haptic output devices 330 may further include inertial or kinesthetic haptic actuators, eccentric rotating mass (“ERM”) haptic actuators in which an eccentric mass is moved by a motor, linear resonant haptic actuators (“LRAs”) in which a mass attached to a spring is driven back and forth, vibrotactile haptic actuators, shape memory alloys, and/or any combination of haptic output devices described above.
The haptic data units 170 are basic units of computer data that may be transmitted to other computing devices via any suitable transmission protocol. The haptic data units 170 may include, for example, segments, packets, and/or frames.
The authoring user interface system 151 is a software module operating on the haptic authoring system 100. The authoring user interface system 151 provides an interface to a user to permit the authoring of haptic effects. The authoring user interface system 151 permits the authoring of a series of haptic effects as a haptic track. The authoring user interface system 151 allows a user to input, design, and/or modify haptic effects through interaction with the authoring user interface system 151. A user interacts with the authoring user interface system 151 through one or more haptic input devices 160 as well as any type of traditional computer input.
The authoring user interface system 151 receives haptic inputs via one or more haptic input devices 160 and generates haptic effect information 171 based on the haptic inputs. The haptic input devices 160 may include basic computer input devices such as keyboards, mice, and touchscreens. The haptic input devices 160 may also include specialized devices configured for reception of input gestures, such as haptically enabled gloves and/or other wearables, cameras, radars, or other sensors configured to capture gestures, and any other device capable of receiving haptic inputs. For example, a user may cause the authoring user interface system 151 to generate haptic effect information 171 by providing input gestures as input via a haptically enabled gloves. In another example, a user may operate a keyboard, mouse, and/or touchscreen to define the parameters of a haptic effect and/or to access a pre-stored library of effects. In embodiments, the authoring user interface system 151 may permit a user to access one or more databases or libraries of haptic effects, from which the user may draw haptic effects when creating a haptic track.
The authoring user interface system 151 further permits a user to modify the haptic effect information 171 of input haptic effects. Haptic effect information 171 includes a length, duration, frequency, type, and/or any other parameter of a haptic effect and is modifiable by a user via the authoring user interface system 151. The user also may use the authoring user interface system 151 to combine haptic effect information 171 of multiple input haptic effects to form combined haptic effects and/or to develop a track of haptic effects. The authoring user interface system 151 may also be used to access or receive previously authored haptic effect information 171 or haptic effects, either for use in generating new haptic effects or for encoding directly into haptics streams. Previously authored haptic effect information 171 may be stored, for example, in a storage location that may include the storage device 120 and/or any other computer memory accessible by the haptic authoring system 100, including additional peripheral storage devices, portable computer memory devices, and networked computer memory devices.
The haptic encoding system 153 is a software module operating on the haptic authoring system 100. The haptic encoding system 153 encodes the haptic effect information 171 received, modified, and/or generated via the authoring user interface system 151. The haptic encoding system 153 accesses the master haptics library 190 to perform the encoding.
The device haptic effects library 290, stored on the haptic server system 200, is a collection of device specific sub-libraries 291.
The client haptics library 390 is stored on the client haptic device 300 and is a full or partial copy of the device specific sub-library 291 that corresponds to the model or type of the client haptic device 300. The client haptics library 390 may be maintained in various ways in different embodiments. As illustrated in
In an embodiment, an individual client haptic device 300 may have a specific customized client haptics library 390 associated with the individual client haptic device. A user may customize the client haptics library 390 according to their own preferences. For example, a user may prefer the sensation when a particular haptic effect is output via the device speaker, rather than a device actuator. That user may create a customized client haptics library 390 that maps the content identifier of the particular haptic effect to their preferred haptic output for their specific device. In such an embodiment, automatic updates from the haptic server system 200 may be disabled.
With continued reference to
In this way, the haptic block template 450 can store information that represents or indicates a basic haptic effect without a requirement to store a control signal or other signal that fully defines the basic haptic effect. For example, consider a basic haptic effect comprising three oscillations of an ERM actuator at a specific frequency and magnitude. This basic haptic effect is fully defined by a sine wave having three full repetitions which may require a relatively significant amount of data to store fully. The basic haptic element ID 451 consists of a shorthand identifier indicative of a three oscillation effect while the block parameters 452 store the frequency and magnitude.
With additional reference to
A more complex haptic effect may be represented by multiple haptic blocks 525. For example, a vibration that requires multiple differing magnitudes or frequencies may be represented by successive haptic blocks 525 having different magnitude parameters or basic haptic element IDs 451 representing basic haptic elements at differing frequencies. Another complex haptic effect may require multiple actuators operating in conjunction, and thus may be defined by multiple haptic blocks 525 defining basic haptic elements to be played by the multiple actuators. Multiple haptic blocks 525 may have the same time stamp 526 if the effects are intended to be played simultaneously.
The haptic encoding system 153 continues to generate additional haptic blocks 525, by selecting haptic block templates, block parameters, and time stamps according to the haptic effect information 171 received via the authoring user interface system 151. In embodiments, the haptic encoding system 153 encodes a stream or track of haptic effect information 171 after it has been completely authored and stored. In embodiments, the haptic encoding system 153 encodes haptic effect information 171 substantially while it is being authored. In embodiments, the haptic encoding system 153 encodes haptic effect information 171 in substantially real time, e.g., as the haptic effect information 171 is being authored or at substantially the same rate as it is authored.
The haptic encoding system 153 is thus able to encode haptic effect information 171 of a series of haptic effects as a successive series of haptic blocks 525. By using haptic blocks 525, which each represent small, simple, portions of larger and more complex haptic effects, a wide array of haptic effects can be encoded. Because the haptic blocks 525 represent the most basic effects of which haptic actuators are capable, a combination of blocks can be selected to define longer or more complex haptic effects that the actuators may be capable of.
The encapsulating system 154 is a software module operating on the haptic authoring system 100. The encapsulating system 154 encapsulates the encoded haptic blocks 525 along with their timestamps into a series of haptic data units for transmission to client haptic devices via a haptic server.
Each session ID 530 designates a complete haptic effect consisting of one or more haptic blocks 525 and corresponding timestamps 526. A complete haptic effect is a haptic effect intended by the haptic designer or author to be output as a single unit. If a portion of a complete haptic effect is not played, then the haptic designers intent for that effect is not met. For example, a complete haptic effect may include a series of vibration effects at different magnitudes punctuated by a final kinesthetic effect. Another example is a crescendo/decrescendo effect wherein a vibration effect increases in magnitude to a peak and then decreases in magnitude until it is complete. If a basic haptic element of the crescendo/decrescendo effect, i.e., a specific magnitude portion of the crescendo, were not output, it would compromise the intended effect. The series of haptic blocks 525 and timestamps 526 required to define the complete haptic effect may be encoded and assigned a session ID 530. Where data unit size limitations force a haptic effect to span multiple haptic data units 170A, 170B, a session ID 530 may be the same across the haptic data units 170A, 170B, permitting a decoder to reconstruct the entire complete haptic effect.
Each access ID 540 represents an access control designating a class of clients with permissions to receive the complete haptic effect designated by the session ID 530. Access IDs 540 are used to selectively distribute haptic effects according to access levels of clients. Access levels of clients may, for example, be based on subscriptions and/or other methods of classification. For example, a live stream artist that provides haptic effects with their video stream may provide one set of haptic effects for casual viewers and a second, more exclusive, set of haptic effects for subscribing viewers. In embodiments, an author of a series of complete haptic effects may designate specific effects to be received based on access identifications. In embodiments, a library or database of complete haptic effects from which a haptic effect author selects effects may include associated access identifications.
Returning now to
The authoring communications system 155 transmits the haptic data units 170A, 170B, 170C, 170D as part of a haptics stream (e.g., master haptics stream 400) to the haptic server system 200 for eventual transmission to client haptic devices 300. The authoring communications system 155 is further configured to transmit an audiovisual stream 375 of audiovisual data to the haptic server system 200, where the audiovisual stream 375 corresponds to the master haptics stream 400. The timestamps 526 of the master haptics stream 400 may correspond to specific points in time of the audiovisual stream 375 at which the haptic effects are intended to be played.
In embodiments, the haptic authoring system 100 also transmits a master haptics sub-library 391 to the haptic server system 200. In embodiments, the master haptics sub-library 391 is transmitted at the outset, when communications are initially established between the haptic authoring system 100 and the haptic server system 200. The master haptics sub-library 391 is a portion or subset of the master haptics library 190 including the content identifiers 191 of all of the haptic block templates 450 that are used by the master haptics stream 400. The master haptics sub-library 391 is used to identify to the haptic server system 200 which haptic block templates 450 are used in the master haptics stream 400. In embodiments, the master haptics sub-library 391 may include all of the information pertaining to the content identifiers 191 and haptic block templates 450 of the master haptics stream 400, including information identifying the basic haptic element ID 451 of the haptic block template 450 as well as information identifying block parameters 452 associated with the basic haptic element ID 451 of the haptic block template. In embodiments, the master haptics sub-library 391 may include only information identifying the content identifiers 191 and haptic block templates 450 used by the master haptics stream 400. The information identifying the haptic block templates 450 is used by the haptic server system to determine the constituent parts of each haptic block template 450 that is used in the master haptics stream 400.
Referring now to
The server communications system 255 is a software module in operation on the haptic server system 200. The server communications system 255 manages all incoming and outgoing traffic from the haptic server system 200. The server communications system 255 is configured to establish and maintain communications with the client haptic devices 300A, 300B, 300C and the haptic authoring system 100. The server communications system 255 may establish communications via a local network, direction connection, and/or via the internet, using any suitable transmission protocol. In embodiments, the server communications system 255 is configured to establish communications via any appropriate network transfer protocol, including, e.g., HTTP, TCP, UDP, etc.
The client identification system 251 is a software module in operation on the haptic server system 200. The client identification system 251 is configured to identify and classify the connected client haptic devices 300A, 300B, 300C to determine a device identification of each client haptic device 300. The client identification system 251 is configured to identify, i.e., determine device identifications, the client haptic devices 300A, 300B, 300C by one or more client device characteristics after communication is established between the client haptic devices 300A, 300B, 300C and the haptic server system 200. Identification of the client device characteristics may be performed, for example, during a handshake process that occurs when communications are initialized. Client device characteristics include access level, capability, client preferences, and other characteristics that identify and define the client haptic devices 300A, 300B, 300C.
The client stream creation system 253 is configured to parse the master haptics stream 400 according to client access levels, as identified by the client identification system 251. As discussed above with respect to
For example, the haptic data unit 170B, as illustrated in
The server communications system 255 is further configured to supply, update, or confirm client haptics libraries 390A, 390B, 390C according to the characteristics of the connected client haptic devices 300A, 300B, 300C, as identified by the client identification system 251. The characteristics of the client haptic devices 300A, 300B, 300C include the haptic capability of the client haptic devices 300A, 300B, 300C, and may be defined by the model number and/or type of client haptic device 300A, 300B, 300C. After communication is established between the client haptic devices 300A, 300B, 300C, and the haptic server system 200, the server communications system 255 ensures that each client haptic device 300A, 300B, 300C has an up-to-date client haptics library 390A, 390B, 390C specific for the model and type of client haptic device. This may include transmitting a client haptics library 390A, 390B, 390C if the client haptic device 300A, 300B, 300C does not have one, updating a client haptics library 390A, 390B, 390C that is out of date, and/or confirming that a client haptics library 390A, 390B, 390C is up-to-date. The structure of the client haptics libraries 390 is discussed above, with respect to
In operation, the haptic server system 200 performs as follows. When a client haptic device 300A, for example, establishes communication with the haptic server system 200 via the server communications system 255, the device characteristics of the client haptic device 300A are identified by the client identification system 251. Based on the identified capabilities of the client haptic device 300A, e.g., the device model or type, the server communications system 255 then either transmits client haptics library 390A to client haptic device 300A, transmits client haptic device 300A an update to the previously stored client haptics library 390A, or confirms with the client haptic device 300A that the previously stored client haptic device 390A is up-to-date. The client identification system 251 identifies the access levels of the client device 300A. The server communication system 255 then transmits the client haptics streams 401A associated with the access level of the client haptic device 300A to the client haptic device 300A.
Returning now to
As illustrated by
As illustrated by
The playback user interface system 351 is a software module in operation on the client haptic device 300. The playback user interface system 351 is configured to provide a user interface 715 for the client haptic device 300. The user interface may be visual, e.g., provided via a display screen or touchscreen, may be tactile, e.g., provided via buttons, switches, or the like, may be audible, e.g., provided via audio output and audio input, and/or may be virtual, e.g., provided via a virtual, augmented, or mixed reality interface. The user interface is configured to permit a user to interact with, e.g., play, pause, stop, select tracks, etc., media content, including haptic content.
The client communications system 355 is a software module in operation on the client haptic device 300. The client communications system 355 manages all incoming and outgoing data traffic from the client haptic device 300. The client communications system 355 is configured to establish and maintain communications with the haptic server system 200. The client communications system 355 may establish communications via a local network, direction connection, and/or via the internet, using any suitable transmission protocol, including, e.g., HTTP, TCP, UDP, etc.
The client communications system 355 is configured to receive the device specific sub-library 291 transmitted by the haptic server system 200 and store the device specific sub-library 291 as the client haptics library 390. The client communications system 355 is further configured to update the client haptics library 390 and/or to confirm that the client haptics library 390 is up-to-date based on the received device specific sub-library 291. The client communications system 355 is further configured to receive the client haptics stream 401, comprising haptic data units 170, from the haptic server system 200.
The decapsulating system 354 is a software module in operation on client haptic device 300. The decapsulating system 354 is configured to decapsulate the client haptics stream 401. Decapsulating the client haptics stream 401 includes removing header and address information from the client haptics stream 401 to extract the haptic blocks 525, along with their timestamps 526 and session IDS 520. The decapsulating system 354 moves the haptic blocks 525 of the client haptics stream 401 to a haptic blocks queue 720 and stores the haptic blocks 525 in the haptic blocks queue 720 according to the session IDs 530. As discussed above, each session ID 530 represents a complete haptic effect to be delivered. The haptic blocks queue 720 stores the timestamps 526 and haptic blocks 525 of each session ID 530 together so that the haptic decoding system 353 can pull information necessary for a complete haptic effect from the haptic blocks queue 720 as required according to the timestamps. In an embodiment the haptic blocks 525 are stored in the haptic blocks queue 720 in the order that they are to be played.
Communications protocols suitable for streaming are often lossy protocols. For example, the UDP protocol does not include any mechanisms for recovering lost data units. When data units are lost or received out of order, they are ignored. This is valuable in a streaming format because it is more important to continue the stream, even with losses, than it is for the transmission to be perfect. In an embodiment, the haptics stream transmission system 10 accounts for potential data unit loss or late arrival by discarding haptic blocks 525 that are received out of order.
The haptic decoding system 353 is a software module in operation on client haptic device 300. The haptic decoding system 353 pulls the collection of haptic blocks 525 associated with each session ID 530 from the haptic blocks queue 720 according to the timestamps of the haptic blocks 525. The haptic decoding system 353 determines the content identifier 191 of each successive haptic block 525. The haptic decoding system 353 accesses the client haptics library 390 to determine the haptic block template 450 associated with the content identifier. The haptic decoding system 353 identifies the basic haptic element identified by the basic haptic element ID 451 associated with the selected haptic block template 450 and applies the block parameters 452 associated with the content identifier 191 to the basic haptic element. The haptic decoding system 353 builds the complete haptic effect associated with each session ID 530 from the basic haptic element IDs 451 and their corresponding block parameters 452, e.g., repetition and magnitude. The haptic decoding system 353 then passes the basic haptic effects defined by the basic haptic element IDs 451 and block parameters 452 of each decoded haptic effect to the haptic playback system 352.
The haptic decoding system 353 of the client haptic device 300 decodes each complete haptic effect according to the local client haptics library 390. Thus, the haptic effects of the client haptics streams 401 are translated to haptic effects which the client haptic device 300 is capable of outputting. Different client haptic devices 300 with equivalent access levels and receiving identical client haptics streams 401 may generate different complete haptic effects based on the specific actuators and capabilities of the different client haptic devices 300. The local client haptics libraries 390, which permit the generation of client specific haptic effects according to universal content identifiers 191 permit the haptic designer to create a single haptic track or stream to be played by many different client haptic devices 300 according to their individual capabilities.
The haptic playback system 352 is a software module in operation on the client haptic device 300. The haptic playback system 352 receives the complete haptic effects as generated by the haptic decoding system 353. The haptic playback system 352 is configured to send or cause the sending of haptic command signals to the haptic output devices 330 of the haptic client device 300 based on the complete haptic effects received from the haptic decoding system 353. The haptic command signals include electrical control signals configured to cause the haptic output devices 330 to output haptic effects according to the complete haptic effects generated for each session ID 530. In embodiments, the haptic playback system 352 is configured to send haptic command signals to the haptic output devices 330 according to the specific capabilities of the haptic output devices 330.
In an operation 902, the process 900 includes receiving, via a processor, haptic effect information. The haptic effect information may be received from a user via a haptic input device and/or may be received from a storage location, such as a local computer memory, a removable or portable computer memory, and/or a networked computer memory. The haptic input device may include traditional computer interface tools such as keyboards and mice used to select parameters of haptic effects via a computer interface. The haptic input device may further include computer interface tools capable of receiving gestures, such as touchscreens, mice, cameras, etc., that permit the haptic effect information to be generated according to input gestures of a user. The haptic effect information includes data describing haptic effects as originally designed by the author of the haptic effects.
In an operation 904, the process 900 includes accessing a master haptics library. The master haptics library is accessed by the processor for the purpose of encoding the haptic effect information. The master haptics library includes associations between content identifiers and designed haptic effects. The designed haptic effects are defined by haptic block templates, which each including a basic haptic element and one or more haptic block parameters to be applied to the basic haptic element, such as magnitude and repetition number.
In an operation 906, the process 900 includes generating haptic blocks according to the haptic block templates of the master haptics library and based on the haptic effect information. Based on the haptic effect information, an appropriate haptic block template 450 is selected from the master haptics library. Based on the haptic effect information, block parameters are determined. The block parameters may include, for example, magnitude and repetition number. Each generated haptic block includes a content identifier that identifies the haptic block template and one or more haptic block parameters. Each haptic block is then timestamped according to the haptic effect information, where the timestamp indicates a time in a haptics stream at which the effects of the haptic block are to be played.
In an operation 908, the process 900 includes encapsulating the haptic blocks.
Encapsulating the haptic blocks includes storing the haptic blocks and time stamps as data units for a haptics stream. Encapsulating the haptic blocks includes identifying all of the haptic blocks of a single complete haptic effect with a session identification. Encapsulating the haptic blocks further includes determining an access identification to associate with each complete haptic effect, where the access identification indicates a client access level required to receive the associated complete haptic effect. The session identifications and access identifications are then encapsulated with the haptic blocks to generate haptic data units. The haptic data units are then provided with the appropriate headers and wrappers for transmission according to a selected transmission protocol.
In an operation 910, the process 900 includes transmitting the haptic data units to a haptic server for distribution. The haptic data units are transmitted as part of a haptics stream for eventual distribution to one or more client haptic devices. In embodiments, an audiovisual stream corresponding to the haptics stream is also transmitted by the haptic authoring system. In such embodiments, the timestamps of the haptics stream are used to synchronize playback of the haptic blocks with the audiovisual stream. In embodiments, the haptic authoring system is configured to generate a master haptics sub-library based on the master haptics library and the selected haptic block templates of the haptics stream, wherein the master haptics sub-library includes the portions of the master haptics library associated with the selected haptic block templates. The master haptics sub-library is transmitted to the haptic server system before, after, or during transmission of the master haptics stream.
The above describes an illustrative flow of an example process 900 of encoding haptic inputs for transmission according to embodiments described herein. The process as illustrated in
In an operation 1002, the process 1000 includes establishing communication between a haptic server system and a haptic authoring system. Communication may be established via any suitable network transmission protocol, including, e.g., HTTP, TCP, UDP, etc.
In an operation 1004, the process 1000 includes receiving from the haptic authoring system a master haptics sub-library. The haptic server system receives a master haptics sub-library from the haptic authoring system. The master haptics sub-library includes a portion of the content identifiers and associated designed haptic effects from a master haptics library. The portion is selected according to the content identifiers used by a master haptics stream to be transmitted from the haptic authoring system to the haptic server system. In embodiments, optionally no library is required to be transmitted between the haptic authoring system and the haptic server system because the haptic server system already stores a copy of the master haptics library. In embodiments, the haptic authoring system and the haptic server system may communicate to determine whether a copy of the master haptics library stored on the haptic server system is up to date.
In an operation 1006, the process 1000 includes receiving a master haptics stream from the haptic authoring system. The master haptics stream is a stream of haptic blocks and timestamps, each associated with a corresponding session identification and client identification. The master haptics stream represents a series of haptic effects as originally designed by the author of the haptic effects.
In an operation 1008, the process 1000 includes establishing communication with a client haptic device. Communication may be established via any suitable network transmission protocol, including, e.g., HTTP, TCP, UDP, etc.
In an operation 1010, the process 1000 includes providing to the client haptic device a device specific haptics sub-library determined according to a device identification of the client haptic device. The device identification of the client haptic device is determined by the haptic server system according to the characteristics of the client haptic device, as discussed above. The haptic server system stores a device haptics library that includes a collection of device specific haptics sub-libraries. Each device specific sub-library is associated with a particular type or model of client haptic device. The device specific haptics library for transmission is selected from the device specific sub-libraries of the device haptics library according to the device identification indicating the model or type of client haptic device. In embodiments, the device specific haptics sub-library sent to a client haptic device is further refined to only include the content identifiers associated with the client haptics stream to be sent to the client haptic device.
In an operation 1012, the process 1000 includes generating one or more client haptics streams from the master haptics stream. The haptic server system parses the master haptics stream into one or more client haptics streams by selecting haptic effects from the master haptics stream according to account identifications of the client haptic devices. The account identifications of the client haptic devices indicate the access identifications that each client haptic device has access to. As discussed above, haptic blocks representing a complete haptic effect are encoded under a session identification and an access identification. The access identification indicates a level of client access required by a client haptic device to receive the complete haptic effect. The level of client access may be identified by an account identification of each client haptic device. The haptic server system constructs one or more client haptics streams for different client access levels by selecting, for each client haptics stream, only those haptic blocks having access identifications corresponding to the appropriate account identification, i.e., level of client access. In embodiments, the haptic server system generates a client haptics stream for each possible level of client access and matches a generated client haptics stream to a client haptic device once the client haptic device is identified.
In an operation 1014, the process 1000 includes transmitting the client haptics stream to the client haptic device. In embodiments, one or more client haptics streams are transmitted to the corresponding one or more client haptic devices. Because the client haptics streams are generated based on client account identifications that indicate access levels and may be the same between multiple client haptic devices, more than one client haptic device may receive the same client haptics stream.
The above describes an illustrative flow of an example process 1000 of transmitting haptics streams to client haptic devices according to embodiments described herein. The process as illustrated in
In an operation 1102, the process 1100 includes establishing communication with a haptic server. The communication link between the client haptic device and the haptic server system may be established according to any suitable transmission protocol including, e.g., HTTP, TCP, UDP, etc.
In an operation 1104, the process 1100 includes obtaining a client haptics library from the haptic server system. The client haptic device supplies the haptic server with a device identification and, in return, the haptic server transmits a client haptics library generated according to the device identification to the client haptic device. The device identification of the client haptic device is determined by the haptic server system according to the characteristics of the client haptic device, as discussed above. The client haptics library is optimized for the client haptic device based on the capabilities of the client haptics device, as identified by the device identification. The client haptics library includes associations between content identifiers and corresponding haptic effects capable of being played by the one or more actuators of the client haptic device. In embodiments, the client haptics library may be generated according to the haptic blocks included within the client haptics stream.
In an operation 1106, the process 1100 includes receiving a client haptics stream from the haptic server. The client haptics stream includes a plurality of haptic blocks and a corresponding plurality of timestamps. Each haptic block includes a content identifier and haptic block parameters. Each haptic block is associated with a session identification. Together, the collection of haptic blocks associated with a given session identification represent a complete haptic effect.
In an operation 1108, the process 1100 includes generating a plurality of haptic effects for playback on one or more actuators of the client haptic device. The haptic effects are generated from the haptic blocks of the client haptics stream. The content identifiers of the haptic blocks serve as an index to the client haptics library to identify the haptic block template called for by the haptic block. The haptic block parameters are applied to a basic haptic element associated with the haptic block template identified by the content identifier. The combination of the basic haptic element and the haptic block parameters provide a basic haptic effect. Combining all of the basic haptic effects decoded from the haptic blocks associated with a given session identification according to their timestamps produces a complete haptic effect.
In an operation 1110, the process 1100 includes outputting the plurality of haptic effects to the one or more actuators of the client haptic device. The client haptic device outputs the generated haptic effects, either in the form of the basic haptic effects or all together in the form of the complete haptic effects, to one or more actuators associated with the client haptic device.
The above describes an illustrative flow of an example process 1100 of receiving haptics streams and playing back the haptic effects encoded therein. The process as illustrated in
Thus, there is provided systems, devices, and methods of encoding, transmitting, decoding, and playing back haptic effects. While various embodiments according to the present invention have been described above, it should be understood that they have been presented by way of illustration and example only, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the invention. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments but should be defined only in accordance with the appended claims and their equivalents. It will also be understood that each feature of each embodiment discussed herein, and of each reference cited herein, can be used in combination with the features of any other embodiment. Stated another way, aspects of the above methods of rendering haptic effects may be used in any combination with other methods described herein or the methods can be used separately. All patents and publications discussed herein are incorporated by reference herein in their entirety.
Additional discussion of various embodiments:
Embodiment 1 is computer implemented method of encoding haptic inputs for transmission, the method being performed on a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method, the method comprising:
Embodiment 2 is the method of embodiment 1, further comprising encapsulating a session identification into the haptic data unit, the session identification representing a complete haptic effect comprising the first haptic block and the second haptic block.
Embodiment 3 is the method of embodiment 2, further comprising encapsulating an access identification into the haptic data unit associated with the session identification, the access identification representing a plurality of clients selected to receive the complete haptic effect.
Embodiment 4 is the method of any of embodiments 1-3, further comprising: transmitting the haptic data unit as part of a haptics stream; and transmitting an audiovisual stream corresponding to the haptics stream.
Embodiment 5 is the method of any of embodiments 1-4, wherein the first haptic block template includes a basic haptic element and at least one block parameter.
Embodiment 6 is the method of any of embodiments 1-5, wherein generating the first haptic block according to the first haptic block template includes selecting a number of repetitions and a magnitude as the at least one block parameter.
Embodiment 7 is the method of any of embodiments 1-6, wherein the haptic effect information is received from a storage location.
Embodiment 8 is the method of any of embodiments 1-7, wherein the haptic effect information is generated according to input gestures of a user.
Embodiment 9 is the method of any of embodiments 1-8, further comprising generating a master haptics sub-library based on the master haptics library, the first haptic block template of the first haptic block, and the second haptic block template of the second haptic block; and transmitting the master haptics sub-library to the haptic server system.
Embodiment 10 is a system for encoding and transmitting a stream of haptic effects, the system comprising:
Embodiment 11 is the system of embodiment 10, wherein the one or more processors are further configured to cause the system to encapsulate a session identification into the haptic data unit, the session identification representing a complete haptic effect comprising the first haptic block and the second haptic block.
Embodiment 12 is the system of embodiment 11, wherein the one or more processors are further configured to cause the system to encapsulate an access identification into the haptic data unit associated with the session identification, the client identification representing a class of clients selected to receive the complete haptic effect.
Embodiment 13 is the system of any of embodiments 10-12, wherein the one or more processors are further configured to cause the system to transmit the haptic data unit as part of a haptics stream, and transmit an audiovisual stream corresponding to the haptics stream.
Embodiment 14 is the system of any of embodiments 10-13, wherein the first and second block parameters include magnitude and repetitions.
Embodiment 15 is the system of any of embodiments 10-14, wherein the basic haptic element comprises an actuator activation of less than approximately 200 ms.
Embodiment 16 is the system of any of embodiments 10-15, wherein the haptic effect information is received from a storage location.
Embodiment 17 is the system of any of embodiments 10-16, wherein the haptic effect information is generated according to input gestures of a user.
Embodiment 18 is the system of any of embodiments 10-17, wherein the one or more processors are further configured to cause the system to generate a master haptics sub-library based on the master haptics library, the first block template of the first haptic block, and the second block template of the second haptic block, and transmit the master haptics sub-library to the haptic server.
Embodiment 19 is a computer implemented method of transmitting a stream of haptic effects, the method being performed on a haptic server having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method, the method comprising:
Embodiment 20 is the method of embodiment 19, wherein generating the client haptics stream includes selecting haptic data units from the master haptics stream according to an account identification of the client haptic device.
Embodiment 21 is the method of embodiment 20, wherein the client haptic device is a first client haptic device, the device specific haptics sub-library is a first device specific haptics sub-library, the device identification is a first device identification, the client haptics stream is a first client haptics stream, and the account identification is a first account identification, the method further comprising: establishing communication with a second client haptic device; providing to the second client haptic device a second device specific haptics sub-library determined according to a second device identification of the second client haptic device; generating the first client haptics stream by selecting haptic data units of the master haptics stream according to a first account identification of the first client haptic device; generating a second client haptics stream by selecting haptic data units of the master haptics stream according to the second account identification of the second client haptic device; and transmitting the second client haptics stream to the second client haptic device.
Embodiment 22 is a haptic server system for transmitting a stream of haptic effects, the haptic server system comprising:
Embodiment 23 is the system of embodiment 22, wherein to generate the client haptics stream, the one or more processors are further configured to cause the system to select haptic data units from the master haptics stream according to an account identification of the client haptic device.
Embodiment 24 is the system of embodiment 23, wherein the client haptic device is a first client haptic device, the device specific haptics sub-library is a first device specific haptics sub-library, the device identification is a first device identification, the client haptics stream is a first client haptics stream, and the account identification is a first account identification, and wherein the one or more processors are further configured to cause the system to establish communication with a second client haptic device, provide to the second client haptic device a second device specific haptics sub-library determined according to a second device identification of the second client haptic device and the master haptic effects sub-library, generate the first client haptics stream by selecting haptic data units of the master haptics stream according to a first account identification of the first client haptic device, generate a second client haptics stream by selecting haptic data units of the master haptics stream according to a second account identification of the second client haptic device, and transmit the second client haptics stream to the second client haptic device.
Embodiment 25 is a computer implemented method of decoding a client haptics stream for playback, the method being performed on a client haptic device having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the client haptic device to perform the method, the method comprising:
Embodiment 26 is the method of embodiment 25, wherein the client haptics library includes associations between content identifiers and corresponding haptic effects capable of being played by the one or more actuators of the client haptic device.
Embodiment 27 is the embodiment of claim 26, wherein the client haptics library is generated according to the haptic blocks included within the client haptics stream.
Embodiment 28 is the method of any of embodiments 25-27, further comprising assigning each of the plurality of haptic effects to the one or more haptic output device for playback according to capabilities of the one or more haptic output devices.
Embodiment 29 is a client haptic device for decoding a client haptics stream for playback, the client haptic device comprising one or more processors; and a storage device storing instructions that, when executed by the one or more processors, cause the client haptic device to establish communication with a haptic server, obtain a client haptic effects library from the haptic server, wherein the client haptic effects library is generated by the haptic server according to a device identification of the client haptic device, receive the client haptics stream from the haptic server, the client haptics stream comprising a plurality haptic blocks and a corresponding plurality of timestamps, generate a plurality of haptic effects for playback on a haptic output device of the client haptic device, each haptic effect selected from the client haptics library according to a corresponding haptic block from the client haptics stream to be played back at a time in accordance with a timestamp of the corresponding haptic block, and output the plurality of haptic effects to the actuator of the client haptic device.
Embodiment 30 is the client haptic device of embodiment 29, wherein the client haptics library includes associations between haptic blocks and corresponding haptic effects capable of being played by the actuator of the client haptic device.
Embodiment 31 is the client haptic device of embodiment 30, wherein the client haptics library is generated according to the haptic blocks included within the client haptics stream.
Embodiment 32 is the client haptic device of claim 29, wherein the one or more processors are further configured to cause the system to assign each of the plurality of haptic effects to the haptic output device for playback according to capabilities of the haptic output device.
