Patent Application
20180013640
This application claims priority from European Application No. 16305866.2, entitled “METHOD, APPARATUS AND SYSTEM FOR RENDERING HAPTIC EFFECTS”, filed on Jul. 8, 2016, the contents of which are hereby incorporated by reference in their entirety.
The present disclosure relates to the domain of immersive haptic effects, for example for obtaining information about a topographical zone through haptic effects.
New technology developments notably focus on haptic stimulation to better immerse users when consuming when living an experience like, for example watching a video, playing a game, attending a festival, skiing or touring a country. The haptic effects allow to share and physically feel what's going on in a virtual environment or audiovisual content. For instance, in “A Framework for Enhancing Video Viewing Experience with Haptic Effects of Motion”, Danieau et al. propose a capture system based on an accelerometer to capture the movement of an actor. This movement can be then played back as a haptic effect of motion on a force-feedback devices.
Thanks to a growing number of wearable devices that people constantly carry on them, it is now possible to record haptic measurements from multiple individuals (e.g. acceleration, position, temperature) spread over a geographical zone. End-users may accept to share the haptic measurements of their wearable device with a community server. However, there is a lack of a system that would allow these end-users to be in turn informed of the state of the haptic atmosphere over a geographical zone via haptic effects.
The purpose of the present disclosure is to overcome the lack of such a system which determine and transmit haptic effects to its end-users' wearable haptic devices according to the state of what this plurality of wearable haptic devices measure over a geographical zone. The present disclosure relates to a method of transmitting data, called second data, representative of a haptic effect, the method comprising:
The present disclosure also relates to an apparatus configured to transmit a second data representative of a haptic effect, the apparatus comprising:
According to a particular characteristic, the received request further comprises a description of the haptic actuators of the client which have sent said request and wherein the second data are further calculated according to said description. Advantageously, the request from the client does not comprise a second location. In this case, the server set the value of the location associated with the last received haptic measurement from said client to the second location.
The present disclosure also relates to a method of rendering a haptic effect on a wearable device equipped with at least one haptic sensor and at least one haptic actuator, the method comprising:
According to a particular characteristic, the request further comprises a description of said at least one haptic actuator. Advantageously, no second location is transmitted in the request, indicating to the server that the second location to use is the location associated with the last sent haptic measurement.
The present disclosure also relates to an apparatus configured to render a haptic effect, said device being equipped with at least one haptic sensor and at least one haptic actuator, the apparatus comprising:
A transmitter configured to transmit, to a server, at least one first data comprising a haptic measurement obtained from said at least one haptic sensor and associated with a first location in a topographical zone;
The present disclosure also relates to a computer program product comprising program code instructions to execute the steps of the method of transmitting data when this program is executed on a computer and also relates to a computer program product comprising program code instructions to execute the steps of the method of rendering a haptic effect when this program is executed on a computer.
The present disclosure will be better understood, and other specific features and advantages will emerge upon reading the following description, the description making reference to the annexed drawings wherein:
The subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. It is understood that subject matter embodiments can be practiced without these specific details.
Haptic technology relates to the sense of touch. Haptic effectors apply forces, vibrations, or motions to the user. Haptic devices may incorporate tactile sensors that measure forces, vibrations, or motions exerted by the user on the interface. In the context of this disclosure, “haptic” has a broader definition that comprises other physical phenomena that a user can feel through her skin and muscles. Effectors and sensors are available for these sensing domains. This includes, for example, temperature (heat or cold effectors and thermometers) and humidity (sprays and hydrometers). A haptic measurement is a data measured by a haptic sensor with this broad definition of haptics or a combination of such data. Reciprocally a haptic effect is an effect that may be rendered by a haptic effector with this broad definition of haptics or a combination of such effects.
According to a non-limitative embodiment of the present disclosure, a method and a server device to collect haptic measurements of a set of wearable haptic devices spread over a geographical area is disclosed. A topographical map of the area is managed by the server device. The map is updated according to the collected haptic measurements. A response is calculated for every client devices that is representative of the haptic atmosphere over the map seen from a given location, for example, the current location of the wearable device at the moment the response is calculated. The response depends on the
According to a particular embodiment, a device takes measurements with its haptic sensors at a regular frequency, for example every one-tenth of a second or every half second or every one second. According to another embodiment, different haptic sensors do not take measurements at the same frequency. A device sends a message to the server each time a new measurement is available from any of its haptic sensors. Such a message comprises, for each available measurement, at least a data representative of the type of the haptic sensor, the value of the measurement and the location at which the measurement has been taken. A message may comprise additional information, for example an identifier of the device or a description of the haptic actuators of the device. In a variant, a device sends a message to a server at a regular frequency, the message comprising the last measurement (and its associated location) of each of its haptic sensors. The regular sending frequency is for example ten hertz or one hertz or 0.5 hertz. In another variant, the location associated with the measurement may be the current location of the device while the measurement is the last one taken by a haptic sensor. In another variant, a device sends a message when one of the value (location or measurement) has changed over a given difference. For instance, a message is sent only if the location is different by at least one meter or one foot from the previous location, or, for example, if the temperature varied at least of half a Celsius degree or one Celsius degree from the previous measurement.
In all embodiments, a device get a response from the server for its participation to the knowledge of the haptic atmosphere over the topographical zone. Another type of client devices equipped only with haptic sensors may be connected to the server. No response is calculated for these client devices as they are not equipped with haptic actuators. Some devices equipped only with haptic actuators may be connected to the server. According to a particular embodiment, such a device sends messages comprising at least its localisation without any associated measurement, and a response may be calculated and transmitted to such a client device despite the absence of its participation to the map.
On the example of
When a user is moving out of the geographical zone, her wearable device is no longer taken into account by the server even if messages from this device keep being received.
According to another embodiment, clusters are calculated according to a weighted Voronoï diagram where points are localisations of the wearable devices and weights, the value of measurements. According to another embodiment, relief 19 or noticeable objects 19 are taken into account for the calculation of clusters. For instance, relief 19 or noticeable objects 19 may facilitate or curb the spread of a cluster over the topographical surface.
Haptic areas are representative of a haptic atmosphere. For example, let's consider a music festival with multiple scenes. A haptic atmosphere is represented by a relevant cluster in proximity of the stages. In front of a metal-rock stage, the haptic data of the cluster allow the server to label this haptic atmosphere as “dynamic”; whereas in front of a love-songs stage, the haptic data of this cluster lead the server to label this haptic atmosphere as “smooth”. Corresponding haptic effects, representative of these semantic labels are calculated to inform the user about each type of atmosphere.
Another example is a football match. A set of spectators spread in the stadium may be equipped with wearable devices and contribute to calculate a map of the different haptic atmospheres over the stands. A TV viewer, watching the game, is then able to choose a camera depending on a preferred haptic atmosphere. Thus, she can feel the atmosphere of a given stand.
The server may use the description of the haptic actuators of the wearable device 15 (in the example of
In another embodiment, the second location sent by the wearable device 15 in its request is different from its own location. For instance, the wearable device 15 is equipped with an interface which allow the user to select a location on the topographical zone 18 and to generate a request according to this selected location. In another embodiment, when the client device already has sent messages comprising haptic measurements, the request may not comprise the second location, indicating to the server to use the location associated with the last sent haptic measurement as the second location.
In accordance with an example, the power source 47 is external to the device. In each of mentioned memory, the word <<register >> used in the specification may correspond to area of small capacity (some bits) or to very large area (e.g. a whole program or large amount of received or decoded data). The ROM 43 comprises at least a program and parameters. The ROM 43 may store algorithms and instructions to perform techniques in accordance with present principles. When switched on, the CPU 42 uploads the program in the RAM and executes the corresponding instructions.
RAM 44 comprises, in a register, the program executed by the CPU 42 and uploaded after switch on of the device 40a, input data in a register, intermediate data in different states of the method in a register, and other variables used for the execution of the method in a register.
The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants (“PDAs”), set-top-boxes and other devices that facilitate communication of information between end-users.
In accordance with an example of rendering haptic effect as illustrated on
In accordance with an example of transmitting data representative of a haptic effect as illustrated in
In accordance with examples, devices 40a and 40b belong to a set comprising:
In an initialization step 51, a client wearable device 40b connects to a server 40a. At this step, a client device inform the server about its participation to the map of haptic atmospheres. In a variant, the client device transmit a description of its haptic sensors and haptic actuators. The server stores these information in order to adapt the calculation of responses to the requests from this device.
The device is then set in state 57. When the device is in state 57, the method is composed of two separated processes which may be executed in parallel or sequentially. On a first hand, in a step 52, measurements are obtained from haptic sensors 48. Actual location of the wearable device is also determined by the positioning module 46. In a step 53, a message comprising the haptic measurement and the actual location of the device is composed and transmitted to the server. When a wearable device 40b is equipped with several haptic sensors, a message is sent each time a new measurement is obtained from the haptic sensors, even if these sensors do not refresh on the same frequency. In a variant, the device microprocessor 42 waits to have obtained a new measurement from each of the haptic sensors before composing a new message. When new measurements are available, they are associated in the message with the last known location. In a variant, the location is updated before a new message is sent.
In a step 54, the wearable device 40b transmits a request to the server. The request comprises the last know location of the wearable device. In a variant, the positioning module 46 checks the actual location of the wearable device before the sending of the request. In another variant, the wearable device 40b is further equipped with an interface which allow the user to select a second location in the topographical zone. This second location is sent in the request. In another embodiment, if the wearable device has already sent at least one message including localized haptic measurement, the request does not comprise a second location indicating that the location of the last sent message has to be considered as the second location by the server. In a step 55, the wearable device waits for a response for a determined delay. When the delay is over and if no response have been received, the device is set back in state 57. If a response to this request is received after the timeout, the response is ignored. As soon as a response has been received, the response is decoded. A response contains the description of haptic effects to render. If the response has been calculated in a generic way, the wearable device may have to convert it in haptic effect renderable with its own haptic actuators. The calculated haptic effects are rendered by the haptic actuators in a step 56. After the rendering is completed, the wearable device is set in state 57.
In an initialization step 61, the map representing the topographical zone that the wearable devices are spread on is initiated. The noticeable objects and reliefs are set up. Each time a new client wearable device is connecting for the first time, this new client is registered. The very first location of the device (at the moment of the registration) is stored as a reference datum. Indeed, the newly connected device may communicate its position by absolute or differential position data. In a variant, the device communicates its own description of haptic sensors and haptic actuators at the registration step.
The device is then set in state 67. When the device is in state 67, the method is composed of two separated processes which may be executed in parallel or sequentially. In a step 62, the server device receives a messages from one of its registered clients comprising haptic measurements associated with a location on the topographical zone, the map is deigned on. In a step 63 this information is added to the known mapped information in order to update the current map. In a variant a brand new map is generated. The sum of haptic measurements from client devices is integrated and clustered according to a clustering function. For example, the clustering function is based on the k-means method. In another variant, the clustering function is based on weighted Voronoï diagrams. Clusters may, for example, be established according to the haptic type of their measurement. The layering of haptic typed maps is forming haptic areas representative of haptic atmospheres.
In a step 64, the server is receiving a request from a client. The request comprises an identification of the sending client and a second location. In another embodiment, the request does not comprise a second location indicating to the server to use the location of the last message sent by this client and received in a step 62 as the second location. In a step 65, the microprocessor 42 of the server device 40a calculates a response adapted to the wearable client which sent the request. The server calculates at least one vector 31 from the location indicated in the request, hereafter called the second location, to one of the haptic area, for example the haptic area, different from the haptic area the second location is contained in, which is the closest from the second location. The distance between the second location and a haptic area is, for instance, calculated as the smallest Euclidian distance on the map between the second location and the edge of the haptic area. In a variant, a centroid is calculated for each haptic area of the map and the distance between the second location and a haptic area is, for example, the Euclidian distance on the map between the second location and the centroid of the haptic area. In a variant, several vectors 31 are calculated from the second location toward the n closest haptic areas for example, with n being equal to two or three or five, for instance. In a variant, the number of calculated vectors 31 depends on the type of haptic actuators the wearable device 15 is equipped with.
In a step 66, the calculated response is sent to the client that the request came from. Then, the device is set in state 67.
Naturally, the present disclosure is not limited to the embodiments previously described. In particular, the present disclosure is not limited to a method of transmitting or of rendering a haptic effect but also extends to any method of adapting a description of a haptic effect to a given apparatus comprising at least one haptic sensor and at least one haptic actuator. The implementation of calculations necessary to generate the first part and subparts is not limited either to an implementation in software programs but also extends to an implementation in any program type, for example shader type microprograms that can be executed by a GPU type microprocessor. The use of the methods of the present disclosure is not limited to a live utilisation but also extends to any other utilisation, for example for recording evolution of calculated map over the time in a long term memory, such as a hard disk.
The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable/personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.
Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and/or depth information. Examples of such equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.
Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD”), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”). The instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two. A processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.
As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.
A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes may be substituted for those disclosed and the resulting implementations will perform at least substantially the same function(s), in at least substantially the same way(s), to achieve at least substantially the same result(s) as the implementations disclosed. Accordingly, these and other implementations are contemplated by this application.
| Number | Date | Country | Kind |
|---|---|---|---|
| 16305866.2 | Jul 2016 | EP | regional |
