Patent Application
20250044871
At least one of the present embodiments generally relates to haptics and more particularly to the definition of a representation format for haptic objects in immersive scenes based on a timeline.
Fully immersive user experiences are proposed to users through immersive systems based on feedback and interactions. The interaction may use conventional ways of control that fulfill the need of the users. Current visual and auditory feedback provide satisfying levels of realistic immersion. Additional feedback can be provided by haptic effects that allow a human user to perceive a virtual environment with his senses and thus get a better experience of the full immersion with improved realism. However, haptics is still one area of potential progress to improve the overall user experience in an immersive system.
Conventionally, an immersive system may comprise a 3D scene representing a virtual environment with virtual objects localized within the 3D scene. To improve the user interaction with the elements of the virtual environment, haptic feedback may be used through stimulation of haptic actuators. Such interaction is based on the notion of “haptic objects” that correspond to physical phenomena to be transmitted to the user. In the context of an immersive scene, a haptic object allows to provide a haptic effect by defining the stimulation of appropriate haptic actuators to mimic the physical phenomenon on the haptic rendering device. Different types of haptic actuators allow to restitute different types of haptic feedbacks.
An example of a haptic object is an explosion. An explosion can be rendered though vibrations and heat, thus combining different haptic effects on the user to improve the realism. An immersive scene typically comprises multiple haptic objects, for example using a first haptic object related to a global effect and a second haptic object related to a local effect.
The principles described herein apply to any immersive environment using haptics such as augmented reality, virtual reality, mixed reality or haptics-enhanced video (or omnidirectional/360° video) rendering, for example, and more generally apply to any haptics-based user experience. A scene for such examples of immersive environments is thus considered an immersive scene.
Haptics refers to sense of touch and includes two dimensions, tactile and kinesthetic. The first one relates to tactile sensations such as friction, roughness, hardness, temperature and is felt through the mechanoreceptors of the skin (Merkel cell, Ruffini ending, Meissner corpuscle, Pacinian corpuscle). The second one is linked to the sensation of force/torque, position, motion/velocity provided by the muscles, tendons and the mechanoreceptors in the joints. Haptics is also involved in the perception of self-motion since it contributes to the proprioceptive system (i.e. perception of one's own body). Thus, the perception of acceleration, speed or any body model could be assimilated as a haptic effect. The frequency range is about 0-1 KHz depending on the type of modality. Most existing devices able to render haptic signals generate vibrations. Examples of such haptic actuators are linear resonant actuator (LRA), eccentric rotating mass (ERM), and voice-coil linear motor. These actuators may be integrated into haptic rendering devices such as haptic suits but also smartphones or game controllers.
To encode haptic signals, several formats have been defined related to either a high level description using XML-like formats (for example MPEG-V), parametric representation using json-like formats such as Apple Haptic Audio Pattern (AHAP) or Immersion Corporation's HAPT format, or waveform encoding (IEEE 1918.1.1 ongoing standardization for tactile and kinesthetic signals). The HAPT format has been recently included into the MPEG ISOBMFF file format specification (ISO/IEC 14496 part 12). Moreover, GL Transmission Format (glTF™) is a royalty-free specification for the efficient transmission and loading of 3D scenes and models by applications. This format defines an extensible, common publishing format for 3D content tools and services that streamlines authoring workflows and enables interoperable use of content across the industry.
Moreover, a new haptic file format is being defined within the MPEG standardization group and relates to a coded representation for haptics. The Reference Model of this format is not yet released but is referenced herein as RM0. With this reference model, the encoded haptic description file can be exported either as a JSON interchange format (for example a .gmpg file) that is human readable or as a compressed binary distribution format (for example a .mpg) that is particularly adapted for transmission towards haptic rendering devices.
Embodiments are related to a data structure storing information representative of an immersive experience comprising a set of haptic effects and a timeline. Haptic effects may be defined in a flexible manner, either directly within the timeline or referenced by an identifier within the timeline. A library may store definitions of the effects with associated identifiers. Data restructuring processes are proposed to convert from a streaming-friendly format where the library is not used to an edition friendly format with the use of a library, and vice versa.
A first aspect of at least one embodiment is directed to method comprising-generating haptic data comprising information representative of a type of effect, a list of haptic effects, a list of haptic tracks, wherein a haptic track comprises a timeline wherein at least one temporal reference is associated to a haptic effect or to an identifier in the list of a haptic effects, wherein a haptic effect comprises information representative of values of a signal to be applied to render the haptic effect.
A second aspect of at least one embodiment is directed to a method for rendering haptic data comprising obtaining haptic data comprising information representative of a type of effect, a list of haptic effects, a list of haptic tracks, wherein a haptic track comprises a timeline wherein at least one temporal reference is associated to a haptic effect or to an identifier in the list of a haptic effects, wherein a haptic effect comprises information representative of values of a signal to be applied to render the haptic effect and providing values of the signal to haptic actuators.
A third aspect of at least one embodiment is directed to a haptic rendering device comprising a processor configured to obtain haptic data comprising information representative of a type of effect, a list of haptic effects, a list of haptic tracks, wherein a haptic track comprises a timeline wherein at least one temporal reference is associated to a haptic effect or to an identifier in the list of a haptic effects, wherein a haptic effect comprises information representative of values of a signal to be applied to render the haptic effect and provide values of the signal to haptic actuators.
A fourth aspect of at least one embodiment is directed to haptic data comprising information representative of a type of effect, a list of haptic effects, a list of haptic tracks, wherein a haptic track comprises a timeline wherein at least one temporal reference is associated to a haptic effect or to an identifier in the list of a haptic effects, wherein a haptic effect comprises information representative of values of a signal to be applied to render the haptic effect.
A fifth aspect of at least one embodiment is directed to a method for restructuring haptic data generated according to the first aspect wherein at least one temporal reference is associated to an identifier in the list of a haptic effects, the method comprising obtaining haptic data, analyzing the haptic data to determine identifiers of a haptic effects in timelines, replacing determined identifiers by copies of the haptic effects associated to the identifiers, emptying the list of haptic effects and providing the restructured haptic data.
A sixth aspect of at least one embodiment is directed to a method for restructuring haptic data generated according to claim 1 wherein at least one temporal reference is associated to a haptic effect, the method comprising obtaining haptic data, analyzing the haptic data to determine identical haptic effects, inserting a copy of one of the determined identical haptic effect into the list of haptic effects, associating an identifier to the inserted identical haptic effect, replacing identical haptic effects by the associated identifier, and providing the restructured haptic data.
According to a seventh aspect of at least one embodiment, a computer program comprising program code instructions executable by a processor is presented, the computer program implementing at least the steps of a method according to the first or second aspect.
According to a eighth aspect of at least one embodiment, a computer program product which is stored on a non-transitory computer readable medium and comprises program code instructions executable by a processor is presented, the computer program product implementing at least the steps of a method according to the first or second aspect.
The haptic rendering device 100 comprises a processor 101. The processor 101 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor may perform data processing such as haptic signal decoding, input/output processing, and/or any other functionality that enables the device to operate in an immersive system.
The processor 101 may be coupled to an input unit 102 configured to convey user interactions. Multiple types of inputs and modalities can be used for that purpose. Physical keypad or a touch sensitive surface are typical examples of input adapted to this usage although voice control could also be used. In addition, the input unit may also comprise a digital camera able to capture still pictures or video. The processor 101 may be coupled to a display unit 103 configured to output visual data to be displayed on a screen. Multiple types of displays can be used for that purpose such as a liquid crystal display (LCD) or organic light-emitting diode (OLED) display unit. The processor 101 may also be coupled to an audio unit 104 configured to render sound data to be converted into audio waves through an adapted transducer such as a loudspeaker for example. The processor 101 may be coupled to a communication interface 105 configured to exchange data with external devices. The communication preferably uses a wireless communication standard to provide mobility of the haptic rendering device, such as cellular (e.g. LTE) communications, Wi-Fi communications, and the like. The processor 101 may access information from, and store data in, the memory 106, that may comprise multiple types of memory including random access memory (RAM), read-only memory (ROM), a hard disk, a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, any other type of memory storage device. In embodiments, the processor 101 may access information from, and store data in, memory that is not physically located on the device, such as on a server, a home computer, or another device.
The processor 101 may be coupled to a haptic unit 107 configured to provide haptic feedback to the user, the haptic feedback being described in the haptic data 192 that is related to the scene description 191 of an immersive scene 190. The haptic data 192 describes the kind of feedback to be provided according to the syntax described further hereinafter. Such description file is typically conveyed from the server 180 to the haptic rendering device 100. The haptic unit 107 may comprise a single haptic actuator or a plurality of haptic actuators located at a plurality of positions on the haptic rendering device. Different haptic units may have a different number of actuators and/or the actuators may be positioned differently on the haptic rendering device.
The processor 101 may receive power from the power source 108 and may be configured to distribute and/or control the power to the other components in the device 100.
The power source may be any suitable device for powering the device. As examples, the power source may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), and the like), solar cells, fuel cells, and the like.
While the figure depicts the processor 101 and the other elements 102 to 108 as separate components, it will be appreciated that these elements may be integrated together in an electronic package or chip. It will be appreciated that the haptic rendering device 100 may include any sub-combination of the elements described herein while remaining consistent with an embodiment. The processor 101 may further be coupled to other peripherals or units not depicted in
Typical examples of haptic rendering device 100 are haptic suits, smartphones, game controllers, haptic gloves, haptic chairs, haptic props, motion platforms, etc. However, any device or composition of devices that provides similar functionalities can be used as haptic rendering device 100 while still conforming with the principles of the disclosure.
In at least one embodiment, the device does not include a display unit but includes a haptic unit. In such embodiment, the device does not render the scene visually but only renders haptic effects. However, the device may prepare data for display so that another device, such as a screen, can perform the display. Example of such devices are haptic suits or motion platforms.
In at least one embodiment, the device does not include a haptic unit but includes a display unit. In such embodiment, the device does not render the haptic effect but only renders the scene visually. However, the device may prepare data for rendering the haptic effect so that another device, such as a haptic prop, can perform the haptic rendering. Examples of such devices are smartphones, head-mounted displays, or laptops.
In at least one embodiment, the device does not include a display unit nor does it includes a haptic unit. In such embodiment, the device does not visually render the scene and does not render the haptic effects. However, the device may prepare data for display so that another device, such as a screen, can perform the display and may prepare data for rendering the haptic effect so that another device, such as a haptic prop, can perform the haptic rendering. Examples of such devices are desktop computers, optical media players, or set-top boxes.
In at least one embodiment, the immersive scene 190 and associated elements are directly hosted in memory 106 of the haptic rendering device 100 allowing local rendering and interactions.
Although the different elements of the immersive scene 190 are depicted in
One advantage of this solution with regards to the structure is that the signal data is easy to package and particularly convenient for streaming purposes Indeed, with such linear structure, the data can be easily broken down to small consecutive packages and does not require complicated data-pre-fetching operations. The signal is easily reconstructed by patching the packages back together to ensure a smooth playback of the signal.
However, with this approach, a part of the signal that is repeated multiple times must be stored multiple times. This linear representation of the data might be convenient for streaming purposes but may be counter-productive for other purposes as it might result in a consequent increase in data size. Additionally, when creating the signal, the haptic designer has to manually duplicate the same haptic effect multiple times which is cumbersome especially in case of modifications of the signal: the modification also needs to be duplicated or redone. Such approach is clearly unsatisfying for an interchange format used in the edition stage of the haptic file.
In addition, the frequency band decomposition for a given track is constant. It is not possible to divide a track in multiple parts with different band decomposition. This limits the possibilities from a designing perspective. For a given track (usually associated to a single haptic actuator), it may be useful to define different effects using different frequency ranges; simple effects might only require a single frequency band, while other effects could be decomposed in multiple layers.
Embodiments described hereafter have been designed with the foregoing in mind.
Embodiments introduce the notion of a haptic effect library allowing to define a haptic effect once and use it multiple times. Following types of haptic effects are defined:
In a first embodiment, basis effects are defined based on the frequency decomposition in haptic bands introduced above. Haptic tracks are then composed of a set of timed haptic effects instead of a list of haptic bands. This solution allows to define haptic effects with different frequency bands within a track. This allows more flexibility to compress the signal by using different frequency bands which may increase the quality of the encoded signal or the compression level. This also offers great flexibility when designing the signal: haptic effects can be encoded using the most relevant band decomposition.
In a second embodiment, basis effects are defined based on keyframes. This solution allows to maximize the space optimization by storing lower level information that can be referenced more easily. With this solution, haptic effects defined in a given haptic band are not depending on the content described in another band.
Both embodiments allow to optimize the space consumption of haptic files. By defining haptic effects in an effect library and using the proposed referencing system, unnecessary data repetition can be avoided through data refactoring.
As illustrated in
For instance, given an existing haptic file with multiple references to haptic effects in the library, the data can be reorganized by replacing the references with the actual effects. This linearization of the data makes the packaging of the data straightforward for streaming purposes. On the opposite, if a haptic file was designed for streaming purposes, with an empty effect library. Parsing the data to identify and extract identical effects to place them in the library could reduce drastically the size of the file.
Such optimization may be introduced in the “binary compression” step 730 according to
The haptic description file (in other words, the haptic data) 300 comprises a first level 301 comprising a file description 302, a set of avatars 303, a set of signals 310 and a shape 305. The file description 302 comprises some generic metadata including the version of the file, the date and a description of the associated haptic experience. The set of avatars 303 comprises the definition of body models on which a haptic effect is to be applied. The shape 305 determines the volume where the haptic effect is active within an immersive scene. Signals such as the signal 311 are aggregated into the set of signals 310. Despite its name, a signal is a relatively high-level concept that ties together haptic effects of similar categories (for example kinesthetic, temperature-based effect, etc.).
The signal 311 is expanded in the second line of the figure and comprises metadata 321, a reference 322 to an avatar selected in the set of avatars 303 (for example a reference to avatar 304), an effect library 330 and a set of haptic tracks 340. The metadata 321 comprises information on the type of haptic effect and corresponding signal (Vibration, Pressure, Temperature, etc.) as well as a description of the signal. The effect library comprises a set of effects such as effects 331 and 332. The example of effect 331 is expanded in the lower left-hand corner of the figure. The set of haptic tracks 340 aggregates several haptic tracks such as 341, 342 and 343. The haptic track 342 is also expanded in the middle-right of the figure.
The effect 331, according to the first embodiment, is defined based on the frequency decomposition in haptic bands. It comprises, at a first level, a set of bands 380, each band corresponding to a range of frequencies, for example the band 381. Each band comprises, at a second level, a set of streams 385, each stream corresponding to a portion of the signal in a given time frame and in a given frequency range. Each stream comprises, at a third level, a set of keyframes 350 (defined by a frequency and an amplitude). For example, the stream 386 comprises at least the keyframes 351, 352 and 353. These keyframes carry the signal that will be applied to the appropriate haptic actuator to finally render the haptic effect, similarly to the streams 271, 272, 273 and 274 of
The haptic track 342 comprises a track description 361 carrying semantic information, track properties 362 carrying some properties such as the gain, the mixing weight, a body part mask and an effect timeline 370. The effect timeline 370 comprises a set of referenced effects associated with a temporal reference such as a timestamp, for example the effects 371, 372, 373 and 374. For the temporal aspects, a timestamp may be associated to each effect. The decomposition of the effect 371 in the bottom-right corner of the figure shows that its structure is identical to an effect defined in the effect library, such as the effect 331.
With such data structure, the designer of the haptic file may choose, for each effect, if it is worth adding it to the effects library to further reference it from the effect timeline or to insert it directly in the effect timeline. Both alternatives are illustrated in the effect timeline 370: the effect 371 (probably used only once) is defined directly within the effect timeline, while the effects 372, 373 and 374 reference to effects from the library, respectively to the effect 331, 331, and 332. With this technique, the effect 331 that is used multiple times is only defined once, thus allowing to minimize the bandwidth and storage space required for the haptic file description.
As seen above, haptic effects can be of three types: Basis, Timeline or Reference. A basis effect is defined by a list of haptic bands describing a signal in the form of streams of waves (or keyframes). It is identified with an id. An example of basis effect is the effect 331 in the effect library. A timeline effect is also identified by an id, but it is defined with a timeline composed of timed effect. An example of timeline effect is the effect 371 in the effect timeline 370. Finally, reference effects are simply references to existing effects defined in the effects library. For this type of effect, the id attribute indicates the effect being referenced. An example of reference effect is the effect 373 in the effect timeline.
An example of JSON schema of a haptic effect according to the first embodiment is given in table 1.
The notion of haptic effect is introduced in the format. At the signal level, effects are stored as a list of effects in an “effect_library”. An example of JSON schema for a haptic signal is provided in table 2.
Effects defined at the signal level can then be referenced and used multiple times in multiple tracks. The effects are organized in each haptic track with a timeline as shown in the JSON schema of table 3.
The table 4 gives an example of JSON syntax for a haptic file according to the first embodiment. In this example, the haptic file contains a single signal with a single track. The effect library contains one basis effect that is referenced multiple time in the track. The track also directly defines another haptic effect. With this solution, one track can contain haptic effects with different frequency bands.
The haptic description file 400 (in other words, the haptic data) comprises a first level 401 comprising a file description 402, a set of avatars 403, a set of signals 410 and a shape 405. The file description 402 comprises some generic metadata including the version of the file, the date and a description of the associated haptic experience. The set of avatars 403 comprises the definition of body models on which a haptic effect is to be applied. The shape 405 determines the volume where the haptic effect is active within an immersive scene. Signals such as the signal 411 are aggregated into the set of signals 410.
The signal 411 is expanded in the second line of the figure and comprises metadata 421, a reference 422 to an avatar selected in the set of avatars 403 (for example a reference to avatar 404), an effect library 430 and a set of haptic tracks 440. The metadata 421 comprises information on the type of signal (Vibration, Pressure, Temperature, etc.) and a description of the signal. The effect library comprises a set of effects such as effects 431 and 432. The example of effect 431 is expanded in the lower left-hand corner of the figure. The set of haptic tracks 440 aggregates several haptic tracks such as 441, 442 and 443. The haptic track 441 is also expanded in the middle-right of the figure.
The effect library 430 is comprising a set of effects such as 431 and 432, defined as a set of keyframes representing values of the signal to be rendered. For example, the effect 431 is defined as composed of keyframes 451, 452, 453.
The haptic track 441 comprises a track description 461 carrying semantic information, track properties 462 carrying some properties such as the gain, the mixing weight, a body part mask and a set of haptics bands 470 such as 471, 472, 473. The example of haptic band 471 comprises some properties 475 and an effect timeline 480. The effect timeline comprises four elements 481, 482, 483 and 484. The elements 481 and 484 are directly defining the effect through a set of keyframes within the timeline while the elements 482 and 483 are referencing effects from the effect library, respectively the effects 431 and 432. Therefore, the element 482 is referencing the keyframes 451, 452, 453 through referencing the effect 431.
An example of JSON schema of a haptic effect according to the second embodiment is given in table 5.
As seen above, haptic effects can be of three types: Basis, Timeline or Reference. A basis effect is defined with an id and comprises a list of haptic “keyframes”. When needed, a phase attribute may be added to the basis effect (not depicted in the figure). An example of basis effect is effect 331 in
At the signal level, the set of effects are stored in the “effect_library” as illustrated in the example of JSON schema of table 6.
The effects are used at the haptic band level, as shown in table 7.
The table 8 gives an example of JSON syntax for a haptic file according to the second embodiment, based on the same example than for the first embodiment (table 4). With this solution, basis effects are easier to define but then every effects of a given track uses the same frequency band decomposition.
Metadata is extracted 710 from the metadata file 701, allowing to identify the descriptive files and/or signal files. Descriptive files are analyzed and transcoded in step 711. In step 712, signal files are decomposed in frequency bands and keyframes or wavelets are extracted in step 713. The interchange file 704 is then generated in step 720, in compliance with the data format according to one of the embodiments described herein. This file may be compressed in step 730 to be distributed in a transmission-friendly form, more compact than the interchange file format.
Other types of optimizations 804 could eventually be added to this process. For instance, instead of only merging identical effects, the process could merge effects that are similar enough to further optimize the gain in space. Such optimization would require additional computing during the signal analysis to evaluate some distance metrics between the effects. Associated distance thresholds would have to be provided to define the degree of similarity between effects to perform the merging.
This data restructuration process could be implemented in multiple manners. For example, it could be inserted after the formatting step 720 in the example architecture of
The table 9 gives an example of JSON syntax for a haptic file according to the second embodiment after a data linearization process. Indeed, the two examples of tables 4 and 8 are already optimized to save space and make the editing of haptic signals easier. For streaming purposes, this data organization is not convenient. The following table shows the same example after linearization of the data. The process replaces every reference to an effect by a copy of the effect and removes the original reference from the library. Based on the second embodiment, the data restructuration results in the following file, much longer, but easier to stream.
In this document, we illustrate the proposed data structure with the JSON readable format, but the same principles apply to the binary version of the format as used in a distribution file.
Although different embodiments have been described separately, any combination of the embodiments together can be done while respecting the principles of the disclosure.
Although embodiments are related to haptic effects, the person skilled in the art will appreciate that the same principles could apply to other effects such as the sensorial effects for example and thus would comprise smell and taste. Appropriate syntax would thus determine the appropriate parameters related to these effects.
Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, mean that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.
Additionally, this application or its claims may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.
Additionally, this application or its claims may refer to “obtaining” various pieces of information. Obtaining is, as with “accessing”, intended to be a broad term. Obtaining the information may include one or more of, for example, receiving the information, accessing the information, or retrieving the information (for example, from memory or optical media storage). Further, “obtaining” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.
It is to be appreciated that the use of any of the following “/”, “and/or”, and “at least one of”, for example, in the cases of “A/B”, “A and/or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and/or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.
| Number | Date | Country | Kind |
|---|---|---|---|
| 21306693.9 | Dec 2021 | EP | regional |
| 22305295.2 | Mar 2022 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2022/081123 | 11/8/2022 | WO |
