Patent Application
20230076264
The present invention is comprised within the field of system and methods for rendering tactile interactions using ultrasound transducer arrays.
Ultrasound haptics produces direct touch sensations on the skin in mid-air (i.e., without the need to hold or wear a haptic device), employing an array of ultrasound transducers as actuators which generate high-frequency pressure waves in the space around the device. Ultrasound phased arrays achieve mid-air stimulation through a phenomenon known as acoustic radiation pressure. Multiple ultrasound transducers, producing an ultrasound wave of the same frequency, are modulated in phase to achieve maximal combined pressure intensity at a certain location in space, known as a focal point. The focus of the ultrasound wave is determined by its wavelength (e.g., 8.5 mm for 40 kHz ultrasound).
By modulating the activation of the transducers it is therefore possible to aggregate the pressure waves at specific points in space, and thus create pressure focal points. Pressure reaches perceivable values at such focal points, and produces a touch sensation in mid-air. Pressure waves produce a complex mechanical interaction on skin, both in space and time, and this mechanical interaction produces an activation and aggregation of mechanoreceptor signals to form tactile percepts. Interacting with fluids in a virtual environment is an interesting phenomenon to be rendered using ultrasound haptics, as they can be moved around without constraining the user's motion.
Iwamoto et al. [1] first demonstrated the capability to use phased array ultrasound to focus sufficient acoustic radiation pressure to induce a haptic sensation in a localized area on the hand, exploiting this principle for mid-air haptic stimulation limited to a fixed focal point. Subsequently, Hoshi et al. [2] extended this achievement to vary the focal location over time, generating moving focal points by temporally varying the modulation of the transducers, producing the sensation of a moving stimulus. Although the induced skin deformation is slight, haptic detection is ensured by modulating the intensity of the field at a frequency to which the skin's mechanoreceptors, principally the Pacinian corpuscles, are sensitive, approximately 150 to 250 Hz. Considering the focal point location as stable relative to this frequency range, this method later became known as amplitude modulation (AM).
These early works used a precomputed time-varying solution for ultrasound phases given a desired spatial amplitude distribution. Because these phases must be determined by means of a non-linear quadratic optimization procedure, the calculations were not amenable to real-time control. Long et al. [3] designed an efficient solution that takes advantage of the linearity of the complex-valued eigenproblem associated with the desired focal point intensities.
Korres and Eid [4] pointed out that the efficiency of this solution admits a different display method, in which focal point amplitudes are relatively stable but their location is modulated at much higher rates. This was called spatiotemporal modulation (STM) by Kappus and Long [5], and they showed that such paths could produce recognizable tactile shapes. Frier et al. [6] found that in STM there is some optimal focal point speed (between 5 m/s and 8 m/s) to maximize skin sensitivity. Recently, Frier et al. [7] analyzed the combined influence of spatial and temporal sampling in STM.
Therefore, there are currently two main control methods to command ultrasound devices: amplitude modulation (AM) and spatiotemporal modulation (STM). AM controls the position and pressure intensity of focal points to produce pressure distributions on skin, while STM controls paths of focal points to draw shapes on skin. In both cases, the high-level control of focal points is translated into low-level control of transducer activation patterns through well-established optimization methods [2], [3].
By eliminating the need to hold or wear a haptic device, ultrasound haptics allows rendering interactions with virtual environments to achieve a more immersive and scalable virtual touch experience. To render virtual tactile interactions, previous works simply display contact locations at maximum intensity, either through AM or STM, without considering the force distributions produced in the virtual interactions. In [8] a target pressure field is extracted from the interaction of a virtual hand with a dynamic fluid simulation, and then AM commands are optimized to induce a best-matching pressure field on the user.
However, AM suffers some limitations. Firstly, the intensity of focal points modulates a pressure wave that induces a perceivable vibration on skin (typically at 200 Hz). Moreover, STM cover larger skin areas with higher perceived intensity, by leveraging constructive interference of focal point paths with the skin waves they induce. However, all previous works command STM with focal point paths of constant intensity which do not represent a best match for dynamic interactions. No previous method computes STM paths of varying intensity to best match dynamic interactions. Not surprisingly, STM poses a more complex challenge than AM. While AM rendering can be regarded as a quasi-static problem, STM requires the solution to a spatial and temporal problem.
Therefore, there is a need for a method of rendering tactile interaction using an ultrasound transducer array that accurately represents dynamic interactions produced in an environment (virtual or real) with a varying pressure or force field (e.g., interactions with a fluid producing a dynamic pressure field in a virtual environment).
[1] T. Iwamoto, M. Tatezono, and H. Shinoda, “Non-contact method for producing tactile sensation using airborne ultrasound”, in Proceedings of the 6th International Conference on Haptics: Perception, Devices and Scenarios, ser. EuroHaptics '08. Springer-Verlag, 2008, pp. 504-513.
[2] T. Hoshi, M. Takahashi, T. Iwamoto, and H. Shinoda, “Noncontact tactile display based on radiation pressure of airborne ultrasound”, IEEE Transactions on Haptics, vol. 3, no. 3, pp. 155-165, 2010.
[3] B. Long, S. A. Seah, T. Carter, and S. Subramanian, “Rendering volumetric haptic shapes in mid-air using ultrasound”, ACM Transactions on Graphics (TOG), vol. 33, no. 6, p. 181, 2014.
[4] G. Korres and M. Eid, “Haptogram: Ultrasonic point-cloud tactile stimulation”, IEEE Access, vol. 4, pp. 7758-7769, 2016
[5] B. Kappus and B. Long, “Spatiotemporal modulation for mid-air haptic feedback from an ultrasonic phased array”, The Journal of the Acoustical Society of America, vol. 143, no. 3, pp. 1836-1836, 2018.
[6] W. Frier, D. Ablart, J. Chilles, B. Long, M. Giordano, M. Obrist, and S. Subramanian, “Using spatiotemporal modulation to draw tactile patterns in mid-air”, in Haptics: Science, Technology, and Applications, D. Prattichizzo, H. Shinoda, H. Z. Tan, E. Ruffaldi, and A. Frisoli, Eds. Springer International Publishing, 2018, pp. 270-281.
[7] W. Frier, D. Pittera, D. Ablart, M. Obrist, and S. Subramanian, “Sampling strategy for ultrasonic mid-air haptics,” in CHI Conference on Human Factors in Computing Systems Proceedings. ACM, 2019.
[8] H. Barreiro, S. Sinclair, and M. A. Otaduy, “Ultrasound rendering of tactile interaction with fluids,” in 2019 IEEE World Haptics Conference (WHC), 2019, pp. 521-526.
[9] G. A. Croes, “A method for solving traveling-salesman problems,” Operations Research, vol. 6, no. 6, pp. 791-812, 1958
Ultrasound transducer arrays are capable of producing tactile sensations on the hand, promising hands-free haptic interaction for virtual environments. However, controlling such an array with respect to reproducing a desired perceived interaction remains a challenging problem. The present invention performs a dynamic mapping of virtual interactions to existing control methods of ultrasound devices, namely, the modulation of focal point positions and intensities over time, a method known as spatiotemporal modulation (STM).
In particular, the invention presents an optimization approach that takes into account known perceptual parameters and limitations of the STM method. This results in a set of focal point paths optimized to best reconstruct a dynamic target pressure field (or a dynamic target force field). The present invention performs a path routing optimization for STM, the first method that commands ultrasound STM to render the force distribution resulting from a dynamic virtual interaction. A key aspect of the method is to pose STM rendering as a quasi-static problem, using careful approximations to eliminate the temporal variable on each dynamic rendering update. As a result, given a target pressure field, STM rendering is posed as the computation of focal point paths that produce the best-matching quasi-static pressure field. Then, given a target pressure field, an optimization algorithm computes focal point paths. In an embodiment, the optimization algorithm first initializes paths over the target domain to optimize coverage weighted by pressure intensity. Then, on a finer scale, the optimization algorithm refines the paths to maximize the similarity to the target pressure.
The present invention refers to a computer-implemented method of rendering tactile interaction using an ultrasound transducer array. The method comprises receiving a target field, wherein the target field is a force field or a pressure field; computing one or more paths of pressure focal points of varying intensity such that the difference between a result field produced by the one or more paths of pressure focal points and the target field is minimized; and controlling an ultrasound transducer array in spatiotemporal modulation mode to simultaneously render the one or more computed paths of pressure focal points. Therefore, the method determines one or more paths of pressure focal points such that the result field produced by these paths is a best-match to the target field.
In an embodiment, the step of computing one or more paths of pressure focal points comprises obtaining one or more initial paths of pressure focal points and optimizing the one or more initial paths of pressure focal points by minimizing a set of cost terms based on the target field and the result field produced by the one or more paths of pressure focal points.
The initial paths are preferably computed by applying a cluster analysis on the target points to obtain a set of sample points; finding a path with minimum length that traverses all the points in the set; if the path is too long, split it into subsets; repeat the last two steps until all paths satisfy a maximum length constraint.
The optimization is performed by iterating, starting with the one or more initial paths, paths of pressure focal point with the aim of minimizing the set of cost terms.
The set of cost terms are preferably minimized by iterating steps of gradient descent. The set of cost terms may include any of the following cost terms: an intensity cost term to maximize field intensity, a length cost term to comply with a target path length, an intersection cost term to prevent path samples from getting closer to a fall-off distance, a bending cost term to obtain low-curvature paths, or a combination thereof.
In an embodiment, the result field is obtained by applying a spatial smoothing function to the intensity of the focal points. The spatial smoothing function may be defined as:
wherein p(x) is the intensity of the result field at position x, xi is the closest position to x in a path, pi is the intensity of the target field at position xi, ϕ is a Gaussian function and γ is a heuristic gain.
The method may comprise projecting the target field onto a plane to obtain a plurality of target points defined by a 2D position and a target value. In this case, the method may further comprise undoing the projection to obtain one or more 3D paths of pressure focal points.
In an embodiment, the method further comprises refining the target field by computing the intersection of the received target field with a surface of a body or a part of a body of a user. The method may also comprise tracking the position of the body or the part of the body of the user.
In an embodiment, the target field may be generated from an interaction of a user with a virtual environment (e.g., a dynamic fluid simulation). The method may further comprise computing an interactive simulation and extracting the target field from the simulation. In another embodiment, the target field is generated from an interaction of a user with a real environment which is located remotely from the user location, such as teleoperation (e.g., remotely controlling a sensorized robotic hand) or telepresence.
In an embodiment, each path of pressure focal points is a 3D curve defining a closed path. The pressure focal points preferably traverse the one or more computed paths at a reference speed comprised between 5 m/s and 8 m/s. Each path of pressure focal points preferably satisfies a maximum path length constraint. The number of computed paths of pressure focal points may be limited to a maximum number of paths (e.g., four paths).
The present invention also refers to a system for rendering tactile interaction using an ultrasound transducer array. The system comprises a data processing device configured to perform the steps of the method.
Another aspect of the present invention refers to a computer program product and/or a computer readable storage medium for rendering tactile interaction using an ultrasound transducer array, comprising computer usable program code for, when executed on a processor, performing the steps of the method.
A series of drawings which aid in better understanding the invention and which are expressly related with an embodiment of said invention, presented as a non-limiting example thereof, are very briefly described below.
When using a spatiotemporal modulation (STM) in the activation of the transducers of an ultrasound transducer array, a focal point traverses a path in space. The path P can be formally described as a time-dependent position: P=x(t)∈ 3. The perceived intensity of a focal point path is maximized under constructive interference between the motion of the focal point and the propagation of skin waves (Frier et al. [6]). This happens for focal point speeds between 5 m/s and 8 m/s; therefore, a reference speed v within said margin is selected for the rendering algorithm. In an embodiment, the selected reference speed v is 7 m/s. Furthermore, to ensure constructive interference on the complete path, the paths are preferably designed as closed paths (i.e., the path P is a closed 3D curve).
Frier et al. [6] assumes that the length of the path is given; therefore, the frequency at which the path is repeated cannot be independently controlled, and depends on the traversal speed and the length of the path. According to preliminary experiments, it has been observed that this is acceptable up to a maximum path length. Beyond that length, the frequency at which the path is repeated is too low, and the stimulus is no longer perceived as continuous. To determine the minimum acceptable frequency (i.e., the maximum acceptable path length), circles of different radii at the reference traversal speed of 7 m/s were rendered, finding that a minimum frequency of 50 Hz (i.e., a maximum path length L=140 mm) is a safe bound to ensure that the stimulus is perceived as continuous.
Normally, the haptic devices (i.e., the ultrasound transducer arrays) allow STM of multiple focal points simultaneously. Each focal point can traverse a different path, with all focal points traveling at the reference speed v, and all paths satisfying the maximum length constraint L. From preliminary experiments, the number of simultaneous focal points can be limited to a certain number of simultaneous paths (in an embodiment, to four paths). More focal points may reach larger coverage, but at the price of notable degradation of perceived intensity.
In previous works STM is used to render 3D curves, wherein the intensity of the radiation pressure of the moving focal point is kept constant along such curves. In the present invention a temporally and spatially varying pressure field is rendered, wherein the intensity of the radiation pressure along the path adapts locally to the intensity of the pressure field. A path is characterized with a position-dependent pressure intensity p(x).
As a focal point cycles multiple times through the same position xi, the rendered pressure intensity p(xi)=pi is the same on all cycles. If the path is repeated frequently enough (e.g., at more than 50 Hz), the rendered radiation pressure produces a persistent tactile perception. This is equivalent to applying a time-invariant pressure at each position along the path, with its effective magnitude a fraction of the rendered pressure. With this assumption, during a time window the effective pressure can be considered to be a spatially varying but temporally invariant field, i.e., a quasi-static pressure field.
The effective pressure of STM rendering is not a simple time-average of the rendered pressure. According to an embodiment, a perceptual heuristic is followed to approximate this magnitude: rendering the same stimuli using amplitude modulation (AM) and STM, and asking subjects to tune the gain γ of STM until the peak perceptual intensity is similar. In practice, a gain γ=1.4 is used.
Focal points exhibit a smooth fall-off determined by the wavelength of the ultrasound signal (e.g., 8.6 mm for the 40 kHz of a test ultrasound device). As shown by Hoshi et al. [2], this fall-off can be approximated well by a Gaussian function ϕ. Based on this finding, together with the heuristic gain γ, the effective quasi-static pressure field p(x) produced by a focal point path can be approximated as:
where xi is the closest position to x in the path. The standard deviation σ of the Gaussian fall-off is set to the same value as the wavelength of the ultrasound signal (e.g., 8.6 mm for 40 Khz).
The quasi-static pressure field assumption simplifies the rendering algorithm of the present invention. Given a target pressure field, obtained for instance from a fluid simulation, the focal point paths whose quasi-static pressure field best reconstructs the target field are searched for. To ensure that the quasi-static pressure field assumption is valid, this search preferably fulfills two constraints, namely that each focal point travels at the reference speed v and the length of each path is equal or shorter than the maximum path length L.
Given a target pressure field or a target force field, focal point paths that produce a best-matching quasi-static pressure field are sought by a path routing optimization algorithm.
The method 100 comprises receiving 110 a target field 102. The target field, which is either a force field or a pressure field, may be computed externally to the optimization method (e.g. by an external simulation). The method also comprises computing 120 one or more paths of pressure focal points of varying intensity such that the difference between a result field produced by the one or more paths of pressure focal points and the target field is minimized. Finally, the method comprises controlling 130 an ultrasound transducer array in spatiotemporal modulation mode to simultaneously render the one or more computed paths of pressure focal points.
In an embodiment, depicted in
If the embodiments of
According to an embodiment, the optimization problem can be solved in two steps, at two different resolutions, searching for paths that maximize coverage and integrated pressure intensity subject to the path length constraint. The first step is a coarse step performing a global search (path initialization), while the second step is a fine step performing a local search (path refinement). In another embodiment, only one of these two steps may be necessary. For example, an embodiment may include using just the first step if the accuracy provided is sufficient, or using only the second step if the computational power allows running the fine step directly.
An interactive simulation may be computed to extract a target pressure field (or a target force field). For instance, an interactive fluid simulation can be used [8], tracking the user's hand and modelling the hand as a moving obstacle in a 3D simulation of a gaseous medium. As described in [8], fluid dynamics can be modelled using incompressible Euler equations discretized on a 3D Eulerian grid, with semi-Lagrangian advection and massively parallel Jacobi relaxation for the pressure solve. The fluid simulation can be executed on a GPU for maximum performance.
The method may comprise refining the target field by computing the intersection of the received target field with a surface of a body or a part of a body of a user. To that end, the method may comprise tracking the position of the body or the part of the body of the user. For example, to define the target pressure field the hand may be voxelized and voxel positions are selected such that they are visible from the side of the domain that corresponds to the location of the ultrasound device. To simplify the path optimization problem a plane may be fit to the voxel positions, so that the voxel positions are projected onto the plane, making path optimization a 2D problem. Formally, the target pressure field can be described by a set of pressure target points T defined by 2D positions xi and their corresponding target pressure values p*(xi):
T={(xi ∈ 2, p*(xi))}.
The first coarse step includes the path initialization. To ensure high computational performance, the initialization of the paths preferably uses only a representative subset of the pressure target points (although the whole set of pressure target points T may also be considered, requiring higher computational resources).
Given the set 504 of pressure sample points 506, a set of closed paths that visit all the points is sought, subject to the maximum path length L. The optimal solution to this problem may require an arbitrarily large number of paths; however, as previously noted, it is advisable to limit the number of paths to a certain number of paths (e.g., four paths). Consequently, the resulting paths may fail to visit all the pressure sample points, and an optimal set must be selected.
This problem is solved iteratively. An optimal path (shortest closed path 508,
The operations to compute an optimal path for a set of points and to split a set of points into further subsets are hereinafter described according to an embodiment, wherein the set of points are either a set 504 or a subset 510 of pressure sample points 506.
Given a set of points, finding the shortest closed path 508 that visits all the points corresponds to the traveling salesman problem. This problem can be solved using the 2-opt algorithm [9], which admits closed paths. The computational cost of 2-opt sets an upper bound on the size of the set 504 of pressure sample points 506. In an embodiment, a maximum of 50 points is set. Therefore, the weighted k-means clustering step is run with 50 or fewer clusters.
To split a set of points, the direction of maximum spread is found, the points along this direction are bounded, and a splitting plane orthogonal to the direction is placed at the midpoint of the two bounds. To find the direction of maximum spread, the covariance matrix of the points, weighted by their pressure value, is computed. The direction of maximum spread corresponds to the eigenvector with highest eigenvalue.
The second fine step includes the path refinement. After initialization, the paths pass through pressure clusters and fulfill the maximum length constraint. However, due to their coarse sampling, the initial paths are not optimally aligned with pressure peaks and ridges. Path refinement is executed at a higher resolution, by up-sampling each path to N points. In an embodiment, N is set to 20, which sets points 7 mm apart from each other, i.e., the distance traveled by a focal point in 1 ms.
During refinement, the goal is to maximize pressure intensity by upsampling each path and moving path samples locally toward locations with higher pressure, while ensuring that paths preserve the following properties: (i) they satisfy the maximum length constraint; (ii) to reach maximum coverage, they do not (self-)intersect; and (iii) they do not bend at sharp angles, as the design decisions of the algorithm stem from perceptual observations on smooth paths, and paths with sharp corners reach smaller coverage. To implement refinement, the goal and the properties are formulated as cost terms of an objective function, and a minimization algorithm is executed.
Given paths with samples {xi ∈2}, we formulate a pressure intensity cost term as:
c
p=−Σip*(xi) (2)
The pressure intensity cost term (cp) is minimized as the path samples xi (i.e., points forming or defining the path) move to locations with higher pressure.
With a target path length L, and N samples per path, the target length is obtained if the length of each path segment is L/N. Then, a length cost term is formulated as:
where xi and xi+1 are two consecutive path samples.
If two paths or two portions of a path get closer than the fall-off distance of focal points, σ, they stimulate overlapping skin areas. The result can be considered inefficient, as the covered skin area is larger if the paths move away. A (self-)intersection cost term that prevents path samples from getting too close is formulated as:
c
i=Σi,j max(σ−∥xj−xi∥, 0)2; (4)
where xiand xj are two non-consecutive path samples.
Finally, to favor low-curvature paths, a bending cost term is formulated as:
where xi−1, xi and xi+1 are three consecutive path samples.
In an embodiment, the paths are optimized by iterating steps of gradient descent of the four cost terms (although fewer cost terms or even additional cost terms may also be considered). For the pressure intensity term, a 2D grid is set with the pressure target values p*(xi), and bicubic interpolation is used to evaluate the pressure at sub-grid resolution and to compute robust gradients. Furthermore, a line search is applied to ensure that the step along the gradient reduces the cost.
To account for the effective magnitude of the pressure field, a heuristic gain γ, as previously explained in equation (1), is preferably incorporated. The rendered pressure of a point on a path is set as pi=γ p*(xi), based on the target pressure field p*.
Once the 2D paths 514 are fully computed, they are lifted back to 3D for STM rendering on the ultrasound transducer array. This is achieved by undoing the projection of the hand voxels.
Each path is 140 mm long and is traversed in 20 ms at 7 m/s. The STM rendering API of an ultrasound transducer array updates every 1 ms a burst of X consecutive focal point positions (e.g., X being 40 in the Ultrahaptics STRATOS Explore USX). Therefore, each path is linearly up-sampled to 800 points spaced 0.175 mm. These points are fed in groups of 40 points to the STM rendering API of the ultrasound transducer array.
In accordance with a further aspect of the present invention there is provided a system for rendering tactile interaction using an ultrasound transducer array. The system comprises a data processing device configured to receive a target field (either a force field or a pressure field); compute one or more paths of pressure focal points of varying intensity such that the difference between a result field produced by the one or more paths of pressure focal points and the target field is minimized; and control an ultrasound transducer array in spatiotemporal modulation mode to simultaneously render the one or more computed one or more paths of pressure focal points (for instance, by computing spatiotemporal modulation commands corresponding to the 3D focal point paths and sending the commands to an ultrasound transducer array). The data processing device may be configured to implement the method as previously defined in any of the different embodiments.
Both the tracking device 630 and the ultrasound transducer array 620 are connected (e.g., via USB) to the computer running a simulation (for instance, a dynamic fluid simulation). The ultrasound transducer array 620 is actuated by a driver circuit 640 controlled by the computer. The system may comprise the driver circuit 640 configured to actuate the ultrasound transducer array 620. The system may also comprise the ultrasound transducer array 620. The system may comprise the tracking device 630 for tracking the position of the body or the part of the body of the user (in this particular case, for tracking the position of the user's hand 602).
The data processing device 610 implements the method as defined in any of the previous figures. The computed phase delays and amplitude values in STM mode are sent from the data processing device 610 to the driver circuit 640. The data processing device 610 may be configured to compute an interactive simulation (e.g., dynamic fluid simulation) and extract the target field 102 from the simulation. In this case, the target field 102 is generated from an interaction of a user's hand 602 with a virtual environment. The computer receives the position of the hand and updates the target field 102 generated on the user's hand 602 based on the hand position in the virtual environment.
The interactions produced during the virtual simulation may be shown on a display device 650, such as a monitor, configured to represent the interactive simulation in a virtual environment 704.
The data processing device 610 computes the target field 102 and the paths 514 of pressure focal points of varying intensity that produce a best-matching pressure field on the user's hand, according to the method described.
In such interaction, haptic perception is dictated by a spatially and temporally varying pressure field on skin, which is used as target for the optimization algorithm.
| Number | Date | Country | Kind |
|---|---|---|---|
| P202030150 | Feb 2020 | ES | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/ES2021/070075 | 2/1/2021 | WO |
