Patent Grant
10536682
This application claims the benefit, under 35 U.S.C. § 365 of International Application PCT/EP2014/053554, filed Feb. 24, 2014, which was published in accordance with PCT Article 21(2) on Sep. 4, 2014 in English and which claims the benefit of European patent application No. 13305220.9 filed Feb. 27, 2013 and European patent application No. 13305980.8 filed Jul. 9, 2013.
The field of the present invention is that of the generation of haptic renderings during the reproduction of an item of audiovisual content. More specifically, the invention relates to the case where the audiovisual content comprises signals making it possible to control actuators generating movements on the body of the spectator.
These days, cinematographic techniques make it possible to enrich the audiovisual content and the experience of the spectator. The advent of 3D offers the spectator a real immersion in the film which is becoming increasingly realistic and impressive. Recently, items of equipment have been added to the seat on which the spectator is seated to give him sensations other than those provided by his sight alone. Thus, seats now move according to several degrees of freedom causing the spectator to experience a movement which is associated with the visual and possibly audio content. For example, in the case of a film shot on the front of a roller coaster car, when a descent is reproduced on the cinema screen, the spectator's seat tips slightly forward. Another example consists in making the cinema seats vibrate intensively from left to right during the reproduction of an earthquake scene.
Audiovisual content with haptic enhancement makes it possible to control actuators which cause the spectator to move during the reproduction of the document. In this way, the spectator is aware of the content not only by sight and hearing, but also by touch. His immersion in the reproduction of the document is thus greater and his impression of experiencing the event more intense.
Most of the time, haptic enhancements are added for films already made. An operator views the film and determines a sequence for which perception would be enhanced for the spectator by a movement performed on his person. The operator manually determines the type of movement (displacement, vibration, throbbing, trembling, etc.) and programs the activation of specific actuators during this sequence. The haptic parameters are added to the video and possibly audio signals on a specific medium. During the reproduction, the haptic parameters are read and transmitted to the actuators responsible for applying stimuli to the spectator. These stimuli can generate all kinds of sensations: well-being, anxiety, assurance, smallness, vertigo, etc.
Currently, these haptic parameters are calculated manually by an operator viewing the video or audiovisual content. There therefore exists a real need for a new technique making it possible to automate the creation of haptic actuator control parameters and the enhancement of new video content.
The present disclosure proposes a new solution which does not have all the disadvantages of the prior art, in the form of a method for reproducing an item of video content filmed using a camera. This method comprises a step of processing the commands applied to the camera during the filming of one or more sequences in order to calculate at least one control parameter for at least one haptic actuator associated with the sequence or sequences and a step of controlling the at least one haptic actuator using the at least one control parameter calculated during the reproduction of the at least one sequence.
In this way, the video content is automatically enhanced by haptic actuator control parameters calculated from commands applied to the camera during filming of the sequence or various sequences of the video content.
According to a first embodiment, the numerical values of the commands applied to the camera produce numerical values controlling the haptic actuators. In this way, the action of the actuators is perfectly adapted to the video content which is reproduced at the same time.
According to another embodiment, a cinematographic effect is determined from commands applied to the camera, and the haptic actuator control parameter is calculated from the cinematographic effect thus determined. In this way, the action of the actuators on the spectator is well adapted to the cinematographic effect which the film-maker wanted to provide in the video content.
According to another embodiment, the video content and control values applied to the camera are transmitted during the sequences to a device for reproducing video content and controlling haptic actuators, the step of processing for calculating the haptic actuator control parameters being executed by this device. In this way, the control parameters for the actuators are well adapted to the installation used for the reproduction of the content.
According to another embodiment, the video content and haptic actuator control parameters are transmitted to a device for reproducing video content and controlling haptic actuators. In this way, the actuator control parameters are calculated only once by the film-maker.
According to another embodiment, a plurality of groups of haptic actuator control parameters are associated with each sequence of the content, each group of parameters being associated with a device for reproducing video content and controlling particular haptic actuators. In this way, the installation used for the reproduction of the content chooses the control parameters according to the haptic actuators available.
According to another embodiment, the duration of a sequence is a multiple of the duration for reproducing an image of the video content. In this way, the temporal division of the content during which a haptic control parameter is calculated is simpler.
According to another embodiment, the actuators are three in number, two actuators intended to be gripped, one in each hand, by a spectator, and one actuator intended to support the head of said spectator. In this way, the installation for reproducing video with haptic feedback is easy to install and relatively inexpensive.
According to another embodiment, if no cinematographic effect is detected during a sequence, then the haptic actuator control parameters associated with this sequence are calculated from control values applied to the camera during filming of this sequence. In this way, the spectator detects haptic sensations even in the absence of detectable cinematographic effects.
According to a first hardware aspect, the present disclosure also relates to a device for reproducing at least one sequence of an item of video content filmed using a camera, the device comprising at least one processor configured to process commands applied to the camera during filming of the at least one sequence in order to calculate at least one control parameter for at least one haptic actuator associated with the at least one sequence and to control the at least one haptic actuator using the at least one control parameter calculated during the reproduction of the at least one sequence.
According to a particular embodiment, the at least one processor is further configured to determine a cinematographic effect from commands applied to the camera, the control parameter for the at least one haptic actuator being calculated from the determined cinematographic effect.
Advantageously, the at least one processor is configured to control three haptic actuators, two actuators being intended to be gripped, one in each hand, by a spectator, and one actuator being intended to support the head of said spectator.
According to another aspect, the present disclosure also relates to a computer program comprising instructions for implementing the method for reproducing video content described above, when the program is executed by a processor.
Other characteristics and advantages of the present disclosure will emerge more clearly upon reading the following description of a particular embodiment, provided as a simple non-restrictive example and referring to the annexed drawings, wherein:
The general principle of the present disclosure lies in a method for reproducing an item of video content filmed using a camera. An item of video content composed of one or more sequences is developed and enhanced by commands applied to the camera at the time of filming. The commands applied to the camera are extracted for each sequence and make it possible to calculate at least one control parameter for one or more haptic actuators associated with this sequence. When the commands applied to the camera cannot be extracted from the video sequence or sequences, these commands are estimated from the video sequence or sequences, as described for example in the article by Thomas G A entitled “Sports Real-Time Camera Pose Estimation for Augmenting Scenes” published in February 2006 during the Conference on Visual Media Production (CVMP) in London. At the time of the reproduction of a sequence of the video content, at least one control parameter thus calculated controls at least one haptic actuator. In this way, the spectator perceives stimuli making it possible to enhance his perception of the video document during the reproduction.
Advantageously, the player able to reproduce the enhanced content determines a cinematographic effect for a set of sequences from commands applied to the camera. Then the haptic metaphor associated with this effect is applied to this set of sequences.
Initially, in step 1.1, an item of video or audiovisual content is filmed. The commands applied to the camera throughout filming are recorded and each group of commands applied to a sequence of given images is associated with this sequence. The content constituted of video data, camera commands and possibly audio data is transmitted to a location where it is reproduced (step 1.2). In step 1.3, a user initiates the reproduction of the content. During the reproduction and for each sequence, the commands applied to the camera are processed in order to calculate at least one haptic actuator control parameter associated with this sequence (step 1.4). Concomitantly, a sequence of the video content is reproduced (1.5) and at least one control parameter thus calculated controls at least one haptic actuator during the reproduction of this associated sequence (1.6).
In step 1.7, the end of the document is tested. If there remain other sequences to be reproduced, the program loops to step 1.4.
The player 1 processes the data of the video content, and calculates the signals to be sent to the reproduction screen 2, the speakers 3 and the set of haptic seats 4. The transmission of the data can be done by cables connecting these various elements or by radio waves. The haptic rendering according to the present disclosure consists in converting the commands which have been applied to the camera during filming of the content into control parameters for controlling the haptic actuators. The actuators cause a movement to a part of the body of a spectator or generate a tactile sensation. The perception of the content which is provided by what the spectator sees and hears is enhanced by what the spectator feels through his body.
The device 8 comprises the following elements, connected to each other by an address and data bus 85 which also transports a clock signal:
It is noted that the word “register” used in the description of memory 83 designates in each of the memories mentioned a memory zone of low capacity (some binary data) as well as a memory zone of large capacity (enabling storage of a whole program or all or part of the data representative of data calculated or to be displayed).
When switched on, the microprocessor 81 loads and executes the instructions of the program contained in the RAM 83.
The random access memory 83 notably comprises:
The algorithms implementing the steps of the method specific to the present disclosure and described below are stored in the RAM 83. According to a variant, the device 8 comprises a graphics card comprising one or more GPUs (graphical processing units) associated with a GRAM (graphical random access memory), the control parameters for the actuator or actuators in this case being stored in the GRAM. According to another variant, the power supply 86 is external to the device 8.
Each haptic actuator 5 and 6 has three degrees of freedom.
A way of calculating at least one haptic actuator control parameter, from commands applied to the camera, will now be detailed using two variants.
The commands applied to the camera 10 are continually recorded during filming and associated with a sequence of a determined duration. These commands can be measurements from displacement sensors (such as gyroscopes) measuring the displacements and the rotations which an operator applies manually to the camera. This determined duration corresponds for example to the time for reproducing an image. For an item of video content reproduced with 24 images per second, the sampling time is 41.66 milliseconds. Each image taken at an instant t is associated with a group C(t) of control values applied to the camera:
C(t)=[xc(t),yc(t),zz(t),ϕc(t),θc(t),ψc(t),γc(t)]
It is possible to use other durations for sequences associated with a group of commands for the camera. These durations are preferably multiples of the time for reproducing an image of the video content. For example, if the duration corresponds to the time for reproducing 10 images (that is to say 416.6 milliseconds), the associated group comprises the total displacement and the total angle of rotation applied to the camera during the filming of these ten images. Each group C(t) of control values is associated with the set of images of a sequence of the content. The video content as it is transmitted for the purposes of reproduction notably comprises the audiovisual data associated with a succession of groups of control values for the camera.
The video document thus enhanced is transmitted by any means to the reproduction location, an example of which is shown in
At the time of the reproduction, the player 1 divides the audiovisual content into sequences associated with a group C(t) of control values applied to the camera; a sequence can contain a single image. Then, the player 1 calculates at least one haptic actuator control parameter associated with this sequence. According to a preferred embodiment, the player controls three actuators each having three degrees of freedom. At any time, an actuator can be represented by a vector representing the position in space of the knob with respect to the base of the actuator. That is to say, given the point GA of the centre of the knob 9 at a given instant t, and the point GA′ at the next instant t+1, the command for controlling an actuator A at a given instant consists in calculating the coordinates of the vector {right arrow over (GAG′A)} using the following equation:
The vectors {right arrow over (T)}, {right arrow over (R)}, {right arrow over (F)} represent respectively a translation, a rotation and a zoom of the image. The vectors {right arrow over (T)}, {right arrow over (R)}, {right arrow over (F)} are defined by the following equations:
As can be seen in the above equation, the vector {right arrow over (T)} depends on three values xc, yc and zc applied to the camera during filming of the video content at an instant t. The vector {right arrow over (R)}, depends on three possible rotations θc, ϕc and φc of the camera at a given instant t and the zoom γc corresponds to a translation F in the forward direction (zoom in) or in the backward direction (zoom out) at a given instant t. Rx, Ry and Rz are 3D matrices about the x, y and z axes of the camera, and I3 is the identity matrix in R3.
The equations above are given only by way of example; they define the movement of the actuator according to the movement made by the camera during filming. The equation describing the function R is particular in the sense that it transforms a rotation of the camera into a translation applied to the vertical shaft of the actuator. This transformation is linked to the position of the actuator in space. The movement is different for each actuator.
For example, if the camera performs a rotational movement θ about the Y axis, the equation for C becomes: C(t)=[0, 0, 0, 0, θ(t), 0, 0].
The vector calculations are:
{right arrow over (GG′A)}=Ry(θc){right arrow over (GGA)}{right arrow over (GAG′A)}={right arrow over (GG′A)}−{right arrow over (GGA)}=(Ry(θc)−I3){right arrow over (GGA)}
The principle is generalised for all the rotations in the preceding equations for the vector {right arrow over (R)},
The values k, m and s are reduction factors ensuring that the position calculated for each vertical shaft 8 does not leave its work space within the actuator.
For example, assume that the camera makes a movement of 2 metres in the forward direction, which corresponds numerically to z=2 m; the equation C then takes the following values.
C(t)=[0,0,2,0,0,0,0]
In the prototype, the work space of a vertical shaft is represented by a 10 cm (0.1 m) cube. The actuator can be moved from −0.05 to +0.05 m along an axis. Thus the camera movement results in a forwards displacement of the actuator of 0.05 m. In this example Kz=0.025 and the equation of the vector {right arrow over (GAG′A)} becomes:
It is noted that the vectors {right arrow over (R)}, and {right arrow over (F)}, are zero in this example; the haptic movement only corresponds to a translation.
If the same control value is applied to the camera during a determined period, this period constitutes a complete sequence during which the maximum amplitude of the movement of the camera is used to calculate factor k.
The other factors m and s are calculated in the same way.
The vectors T, R and F make it possible to control the three degrees of freedom of each actuator. For example, when the camera performs a rotation to the right, then the right-hand actuator moves forward, the left-hand actuator moves back and the head actuator pivots slightly to the right.
The player 1, while sending the video signals to the screen 2 and the audio signals to the loudspeakers 3, calculates the actuator control parameters and transmits them in real time. The set of signals (video, audio and haptic control parameters) are transmitted synchronously so that the movements generated by the actuators correspond to the images and sounds of the associated sequence.
A variant embodiment of the present disclosure consists in analysing the commands applied to the camera during a sequence or a group of sequences in order to determine a cinematographic effect during this sequence and in applying control parameters corresponding to this effect to the various haptic actuators.
Here is an example of a list of cinematographic effects which can be detected by analysing the commands applied to the camera during filming:
This effect consists in rotating around characters to emphasise these characters in a scene. The haptic metaphor consists in moving the knobs of the two hand actuators towards each other in order to give the impression that the spectator is concentrating.
This effect consists in moving the camera back (for a crane shot which tracks out) while raising it above the scene; this effect is often used at the end of a film to give a feeling of moving backwards and soaring above the object filmed. The haptic metaphor consists in applying small tilting movements to the actuators to simulate a wing movement enabling the spectator to soar.
This effect consists in filming a scene while tilting the camera slightly to the left to view a character in an unusual way and make him appear disturbing. The haptic metaphor consists in rocking the knobs of the hand actuators from left to right alternately in order to destabilise the spectator and give him an unsettled feeling.
This effect consists in performing a lateral displacement, in order to show scenery or to follow a moving character. The haptic metaphor consists in simulating a crabwise advance, that is to say quickly moving the right hand away, then quickly moving the left hand away while slowly bringing the right hand back to its original position, then quickly moving the right hand away while slowly moving the left hand back, and so forth.
This effect consists in filming a character either from above or below. A tilt-down shot causes a dwarfing effect. It can be used, among other things, for expressing the point of view of a character or for restoring the subject to being a detail in the shot. A tilt-up shot can be used to increase the significance of the subject in the shot and give the spectator an impression of smallness. According to the subject, the tilt-up shot can be associated with a “natural” point of view: if the Eiffel Tower is filmed using tilt-up shots, a human-scale point of view is adopted. The haptic metaphor consists in bringing the three actuators downwards, the hands and the head tilting forwards.
This effect is so called as it was used in the film “Vertigo” by Alfred Hitchcock. This effect consists in giving the spectator the impression of vertigo. The camera moves forward while zooming out and the object filmed at the centre of the image hardly changes appearance in contrast to what surrounds it. The haptic metaphor consists in moving the knobs apart giving the spectator the strange impression that the scene is opening up.
This effect is the standard zoom-in shot. The haptic metaphor consists in imitating walking by moving the knobs of the two hand actuators forward one after the other.
For each sequence of the video content, the player detects at most one cinematographic effect. If a same effect is present in several consecutive sequences, then these sequences are grouped together in one set and the player transmits, during the reproduction of this set of sequences, the actuator control parameters corresponding to the haptic metaphor associated with this cinematographic effect.
According to an improvement, if no cinematographic effect is detected during a sequence, then the program applies the equations described in the first variants and calculates the haptic actuator control parameters associated with this sequence by directly using the commands applied to the camera during the filming of this sequence.
According to an improvement, the player 1 detects the amplitude of the cinematographic effect for the spectator, and weights the value of the parameters generating the haptic metaphor by amplitude and, if the movement is cyclic, the frequency of the movement applied to the spectator.
The amplitude of a cinematographic effect can be deduced from the control values applied to adjust the camera during filming. Take for example a tilt-up effect (rotation ϕ about the X axis) which corresponds to:
C(t)=[0,0,0,ϕ(t),0,0, 0]
The tilt-up angle determines the amplitude of the cinematographic effect and this amplitude becomes a coefficient for the haptic rendering command. An effect C1 will be greater than an effect C2 if ϕ1(t)>ϕ2(t).
This information can then be used to modulate the power of the cinematographic effect.
According to another example, the following command is associated in order to determine the haptic rendering of the tilt-up, that is to say the vector:
(the angle of rotation moves the actuator A along the Y axis),
In this case, the haptic effect for C1 will be automatically greater than for C2.
According to an improvement applied to the two variants previously described, the content provider calculates the actuator control parameters beforehand and associates them with the sequences of the video content. In this way, the calculation is only performed once and the players 1 responsible for reproducing the content do not need to carry out this processing. This improvement can be used if the content provider knows the characteristics of the actuators used during the reproduction and their positions with respect to the spectators. If the rooms dedicated to the reproduction of an item of enhanced content have different actuator models, then the video document can contain different sets of actuator control parameters, the player retrieving that which is intended therefor during the reproduction.
While the present disclosure was described in reference to particular illustrated embodiments, the present disclosure is in no way limited to these embodiments, but only by the appended claims. It should be noted that changes or modifications to the embodiments previously described can be contributed by those in the profession, without leaving the framework of the present disclosure.
The embodiments previously described are for example implemented in a method or a process, an apparatus, a software program, a data stream or a signal. A device or apparatus implementing the interpolation method described is for example in the form of hardware components, programmable or not, in the form of one or more processors (advantageously of GPU type but also of CPU or ARM type according to variants). The methods described are implemented for example in an apparatus comprising at least one processor, which refers to processing devices in general, comprising for example a computer, a microprocessor, an integrated circuit or a programmable software device. Processors also comprise communication devices, such as for example computers, mobile or cellular telephones, smartphones, portable/personal digital assistants (PDAs), digital tablets or any other device enabling the communication of information between users.
The embodiments of the various processes and various characteristics described previously can be implemented in various equipment or applications, for example notably in an item of equipment or applications associated with the coding of data, the decoding of data, the generation of views or images, texture processing, and any other processing of images or information representative of texture and/or information representative of depth. Examples of such an item of equipment are an encoder, a decoder, a post-processor processing the outputs of a decoder, a preprocessor supplying inputs to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a mobile telephone, a PDA, a digital tablet and any other communication device. The item of equipment can be mobile or on board a mobile vehicle.
Moreover, the methods described can be implemented in the form of instructions executed by one or more processors, and such instructions can be stored on a medium that can be read by a processor or computer, such as for example an integrated circuit, any storage device such as a hard disc, an optical disc (CD or DVD), a random access memory (RAM) or a non-volatile memory (ROM). The instructions form for example an application program stored in a processor-readable medium. The instructions take for example the form of hardware, firmware or software.
| Number | Date | Country | Kind |
|---|---|---|---|
| 13305220 | Feb 2013 | EP | regional |
| 13305980 | Jul 2013 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2014/053554 | 2/24/2014 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2014/131733 | 9/4/2014 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 3469837 | Heilig | Sep 1969 | A |
| 20020105521 | Kurzweil | Aug 2002 | A1 |
| 20020115043 | Baker | Aug 2002 | A1 |
| 20030068053 | Chu | Apr 2003 | A1 |
| 20040054512 | Kim | Mar 2004 | A1 |
| 20050168021 | Real | Aug 2005 | A1 |
| 20060028542 | Rondinelli | Feb 2006 | A1 |
| 20070098268 | Beresford | May 2007 | A1 |
| 20080317386 | Wood | Dec 2008 | A1 |
| 20090158206 | Myllyla | Jun 2009 | A1 |
| 20110044604 | Brokken | Feb 2011 | A1 |
| 20110057941 | Dengler | Mar 2011 | A1 |
| 20110069229 | Lord | Mar 2011 | A1 |
| 20130117798 | Ha | May 2013 | A1 |
| 20130227410 | Sridhara | Aug 2013 | A1 |
| 20140205260 | Lacroix | Jul 2014 | A1 |
| 20160014386 | Danieau | Jan 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 2011198373 | Oct 2011 | JP |
| WO2012116433 | Sep 2012 | WO |
| Entry |
|---|
| G. de Haan and P. W. A. C. Biezen “An efficient true-motion estimator using candidate vectors from a parametric motion model” IEEE tr. on Circ. and Syst. for Video Techn. vol. 8 No. 1 Mar. 1998 pp. 85-91. |
| Christie et al: “Camera Control in Computer Graphics”; Computer Graphics Forum, vol. 27, No. 8, pp. 2197-2218, Dec. 2008. |
| Danieau et al: “A Framework for Enhancing Video Viewing Experience with Haptic Effects of Motion” ;IEEE Haptics Symposium 2012, 2012, pp. 541-546. |
| Danieau et al: “Enhancing audiovisual experience with haptic feedback a survey on HAV-Transactions on Haptics”, 2012; pp. 1-16. |
| Danieau et al: “HapSeat Producing Motion Sensation with Multiple Force-feedback Devices Embedded in a Seat”;in VRST-12, 2012; pp. 1-8. |
| Smith et al: “Communicating emotion through a haptic link Design space and methodology”; International Journal of Human-Computer Studies, vol. 65, No. 4, pp. 376-387, 2007. |
| Thompson: “Grammar of the edit Second Edition”; Focal Press, 2009; pp. 1-225. |
| Thompson: “Grammar of the Shot-Second Edition”, Focal Press, 2009; pp. 1-233. |
| www.d-box.com: Move the World; Jul. 8, 2015; pp. 1-2. |
| Silva et al: “Acoustic-Tactile Rendering of Visual Information”; Proc. of SPIE-IS&T Electronic Imaging, SPIE vol. 8291, 82910M-2012 SPIE-IS&T pp. 1-12. |
| Danieau et al: “Toward haptic cinematography enhancing movie experience with haptic effects based on Cinematographic Camera Motions”; Submitted on Dec. 12, 2013; pp. 1-19. |
| Chang et al: “Overview of the MPEG-7 Standard”; IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, No. 6, Jun. 2001; pp. 688-695. |
| Park et al: “A Framework of sensory information for 4D home theater system—Natural Sciences Publishing Corp”; 2012; Appl. Math Inf. Sci 6, No. 1S, pp. 201S-207S. |
| Rehman et al: “Turn Your Mobile Into the Ball-Rendering live football game using vibration”; IEEE Transactions on Multimedia, vol. 10, No. 6, Oct. 2008; pp. 1022-1033. |
| Tsetserukou et al: “Affective Haptics in emotional communication”, IEEE 2009; pp. 1-6. |
| Search Report dated May 14, 2014. |
| Number | Date | Country | |
|---|---|---|---|
| 20160014386 A1 | Jan 2016 | US |
