Patent Application
20210014631
The present invention relates to a method for determining head-related transfer functions (abbreviated as HRFTs) which are listener-specific to a specific individual.
When sound arrives at the ear drums of a (usually human) listener, the sound is filtered by the head, auricles, and—to a lesser extent—torso of the listener's body. This filtering can be described by HRTFs, which describe the relation between the sound spectrum of the original sound and the sound spectrum of the sound arriving (and perceived) at the ear drums. Generally, HRTFs depend on the spatial position of a sound source and the listener's anatomy; in fact, HRTFs heavily depend on the particular geometry of the body parts, primarily the geometry of the auricles and the head, and thus they are strongly listener-specific. Thus, the HRTFs correlate with the physical characteristics of the individual, primarily the shapes of the individual's head and auricles. In this disclosure the term “listener” is used for an individual for whom an HRTF is to be derived or has been derived. The individual is usually a (specific) human person, but the application may be extended to animals as well, in particular mammals. The term “auricle” or “pinna” denotes the part of the outer ear at the outside of the head, including the entry to the ear canal which is surrounded by the pinna proper.
One use of HRTFs is in filters used for creating spatial audio via headphones, a technique called binaural virtual acoustics (BVA). The challenge of BVA is the acquisition of accurate listener-specific HRTFs.
One method to measure HRTFs is acoustical measurement using microphones in the ear canal of a person, or a replica of the person's head, placing the sound source at various positions, in particular with varying directions, with respect to the person whose HRTFs are to be measured. Measurement of HRTFs is a difficult and resource-demanding procedure, which is often uncomfortable for the listener.
As an alternative to measurements, HRTFs can also be obtained by numerical calculation using a three-dimensional (3D) geometrical model of the listener. Suitable software for calculating HRTFs from a given 3D model are well-known. For acquiring the geometry, prior art used 3D laser scans or silicon impressions scanned in CT devices or scans by means of MRI. These methods are usually not applicable by a consumer at home, where such sophisticated equipment is not available. Other prior art methods used photogrammetric reconstruction based on multiple two-dimensional photographs of the listener for calculating a 3D geometrical representation of the listener. The HRTFs calculated from such 3D models were found to be generally poor in that they did not produce a sufficient spatial sensation with the listener, often causing a so-called “internalisation” (i.e., spurious location of the source as being inside the head), as a result from inaccuracies of the underlying 3D representations.
Furthermore, for checking the validity of an HRTF, prior art methods consider manual analysis by means of visually analysing amplitude spectra. This approach of evaluating an HRTF is not accurate to estimate the actual HRTF quality and is problematic with respect to reproducibility.
U.S. Pat. No. 9,544,706 B1 describes the concept of taking photographs of the pinna, creating a 3D model of the pinna, calculate HRTF, i.e., “a plurality of morphological parameters describing user's pinna” or “determine a customized HRTF”, associate the HRTFs with a user's profile, and apply the HRTFs thus obtained to headphone signals. While this method presents a full chain from the pictures to the binaural signal, it does not provide sufficient details to successfully realize functional HRFTs, in particular with regard to requirements of a valid 3D model and the procedure of deriving the HRTFs from 3D models. Another, similar method for creating HRTFs is described in US 2012/0183161 A1. US 2017/0034641 A1 describes calculating HRTFs from a mesh model representative of head and ear geometry of a listener, obtained from images of the head and the ears taken at various angles.
WO 2017/116308 A1 discloses a method for generating customized HRTFs by taking a photo of the head, determining the scaling factor, taking a second photo of the ear, parametrizing a pinna model by means of the Active Shape Model with manually provided control points, to derive a personalized HRTF based on that model combined with the scaling factor. The Hough transform is used to find a best match between the arcs of the pinna and those from an HRTF database. This method does not describe a calculation of HRTFs, but a matching procedure to an existing HRTF database, which is required a priori.
Therefore, it is an object of the invention to overcome the disadvantages of the prior art. In particular, a method for determining HRTFs is sought which provides a reliable spatial experience of sounds to the listener. Thus, the invention shall provide a methodology for obtaining a sufficiently accurate 3D model of listener's head and auricles suitable for the calculation of HRTFs. Preferably, the method should allow acquisition of HRTFs at home, i.e., without specialized equipment. Further, an evaluation of HRTFs may be suitable for finding modifications of the procedure and/or data to achieve accurate HRTFs.
The above-mentioned aim is met by a method as described in the independent claim 1. In particular, the method comprises the steps of:
The invention is based on the inventors realizing that the 3D model has to fulfil requirements for representing the listener's geometry, where the accuracy of the auricle geometry is essential and has different requirements than the head geometry.
A visual representation of an object of interest is understood as a representation including one or more images of the object of interest obtained through optical means, such as a photographic or video camera (but not restricted thereto), substantially in the visual spectral range.
In particular, the mentioned visual information of anatomical components of the auricle relates not only to the overall shape of the auricle, but also to the shape of typical auricle features. These anatomical components of the auricle may advantageously include the entry of the ear canal, the concha, the fossa, and the backside of the auricle. The inventors noted that the quality of the HRTFs will heavily depend on the presence of sufficient information about the shape of these features in the data of the visual representation and the polygon meshes, respectively; further components of the auricle may be included as well, to further improve the quality/accuracy of the HRTFs. Hereinafter, the term “depiction” is used to denote the parts of the visual representations which relate to the auricles and the head, respectively. Moreover, it is usually advantageous to have the parts of the visual representations which relate to the auricles and the head, respectively, which are hereinafter referred to as depictions.
The visual representation, and more specifically the depictions, may be obtained, in general, from any available source. For instance, the visual representation may be obtained from photographical images, through photographical device(s) and/or from video recording. For instance, the visual representation may be already existing, or it is prepared for the deriving process of HRTFs according to the invention. The depictions of the auricles and the head may be integrated, but preferably they are obtained and processed as separate instances. In the latter case in order to facilitate preparing and handling of the depictions, it may advantageous to prepare the depictions of the auricles separately from each other and from the depiction of the head.
Furthermore, it may be advantageous to operate with separate meshes for the auricles and head, since this can facilitate handling of the different requirements for the auricles and the head. In particular, the requirements of accuracy are much higher for the auricle meshes than for the head mesh. Thus, in this case in step (
In many suitable embodiments of the invention the auricle and head meshes may be composed of a respective plurality of triangles; the polygons or triangles will have an average size in each of the head and auricle meshes. Advantageously, the average size in the head mesh is greater than the average size in the auricle meshes, since this can help reducing calculation time for the head mesh and avoiding unfavourable influences of the head mesh upon the auricle meshes. Depending on the actual application, the average size in the head mesh may be just slightly larger, or by a factor of at least 1.4, preferably at least 2. Moreover, a mesh—i.e., the individual head and auricle meshes or the combined mesh—may be scaled according to at least one distance value which was previously taken at the listener's head and/or auricles, for instance during or at the end of a photo session.
In many preferred embodiments of the invention, depictions are prepared by taking one or more sets of photographs (e.g., through respective “photo sessions”) of the auricles and head of the specific individual, wherein each set includes photographs taken at various orientations of the camera towards the individual's auricle or head. For instance, separate sets may be taken for the auricles and the head, respectively. The photographs may be taken using a suitable photo camera or a video camera. In fact, the visual representation, in particular the depictions or the photograph sets, may be obtained using frames of a video featuring the individual.
During preparing the depictions, such as taking the photographs, it may be advantageous to use a lighting system for illuminating the head and/or auricles. In particular, for an auricle, the lighting system is preferably configured in a suitable manner so as to illuminate specific components of the auricle. For instance, the lighting system may provide lighting into the ear canal, lighting along the navicular fossa, lighting into the lower concha, and/or lighting behind the auricle (auricle backside). Such a lighting system may include a diffuse lighting and at least one additional lighting source, each of said additional lighting source configured to specifically illuminate a respective selected component of the auricle as mentioned. These additional lighting sources may be integrated in the diffuse lighting or realized as separate light sources; on the other hand, a lighting source may be omitted in the case that the diffuse lighting is already sufficient for illuminating the respective feature(s) of the auricles.
Advantageously, the individual may be prepared before preparing the depictions, such as taking the photographs (or generally, preparing the visual representations). Thus, the individual's head and auricles may be prepared to optically reveal the auricles. This treatment may include for each auricle: reversible removing hair from the auricle, e.g., by binding back the hair, combing the hair etc., trimming hair present at the auricle, shaving the complete auricle or selected areas of the auricle.
Furthermore, in step (
It is remarked that it was found that a merely geometric description of the head and auricles is usually sufficient for calculation of good HRTFs, since the skin is treated merely as reflective for the acoustic waves. Therefore, further acoustic properties such as skin impedance, are negligible in the context of calculating HRTFs.
Before step (
In a further aspect of the invention, a methodology is proposed to immediately validate the perceptual relevance of the calculated HRTFs. Thus, the HRTF calculation may be combined with an HRTF evaluation such that parts of the method are iteratively performed until it results in sufficiently accurate HRTFs. A method comprising such evaluation may comprise the additional steps of
Further suitable aspects and benefits of the invention will become evident from the below description of an embodiment of the invention.
In the following, in order to further demonstrate the present invention, illustrative and non-restrictive embodiments are discussed, as shown in the drawings, which show:
The detailed discussion of an exemplary embodiment of the invention given below is intended to illustrate the basic concepts and further advantageous developments of the invention. It will be evident to the person skilled in the art to freely combine several or all of the features discussed in relation to the embodiment and variants discussed here, as deemed suitable for a specific application of the invention. Throughout this disclosure, terms like “advantageous”, “exemplary”, or “preferable” indicate elements or dimensions which are particularly suitable (but not essential) to the invention or an embodiment thereof, and may be modified wherever deemed suitable by the skilled person, except where expressly required. It will be appreciated that the invention is not restricted to the exemplary embodiment, which is given for illustrative purpose, merely representing suitable ways to implement the invention. Furthermore, it will be clear that the reference numbers used in the claims are merely intended to improve the readability of the claims, but neither the reference numbers nor the way they are used in the claims are to be construed as limiting the scope of the claims.
The embodiment of the invention discussed in the following uses photographs taken by a digital camera for deriving HRTFs via separate 3D meshes for the auricles and the head of the listener. Specifically, as illustrated in
In a first step 10, visual representations of the head and auricles are provided. This step includes a preparatory stage and a photo sessions stage: In a preparatory stage 11, the listener, the room, and the camera system are prepared. Then, photo sessions 12a, 12b produce two visual representation of the listener's left and right auricles, respectively, and a third photo session 13 captures the listener's head. The photo sessions 12a, 12b, 13 of the photo sessions stage may be done in any sequence as may be suitable. It will be clear that in other (not shown) embodiments, the visual representation may be taken in one whole photo session. In step 14, 3D meshes of the head, left auricle, and right auricle are calculated based on the visual representations, and these meshed are merged into a single joint 3D object (“combined mesh”) in step 15. Step 16 prepares the combined mesh for the subsequent numeric calculations, which are done in step 17 to obtain HRTFs.
Optionally, step 18 evaluates the HRTF quality; depending of the result obtained from step 18 the steps 16 to 18 may be repeated iteratively until HRTFs of sufficient quality or accuracy are obtained. The operations carried out in the individual steps and stages according to the embodiment of the invention are described in detail below. Where the discussion refers to only one of the ears/auricles, it is understood that the same considerations apply mutatis mutandis for the other ear, by virtue of the principal mirror symmetry between the two ears.
Referring to
In a first stage 11 the listener is prepared, which serves to ensure good visibility of the auricles and suitable surface quality of the skin. The listener uses a nose-and-ear-hair trimmer to trim all hair on the auricles. In case that head hair might obstruct the visibility of parts of the auricles, ribbons or like accessories are used to pull back head hair such that both auricles are visible (i.e., not covered or obscured by hair) from all angles.
As a protection for the ear canals, earplugs are inserted into the left and right ear canal, respectively. Each earplug is made of a disposable foam and has substantially cylindrical shape of a length of approx. 1 cm; in the centre of the outward-oriented base face of each earplug, a hollow is cut out and marked with a black pen; this marking serves to indicate the position of the earplug, so as to be able to identify the location of the entry to the ear canal in later steps. The earplug thus prepared is placed into the respective ear canal such that it closes the ear canal but does not protrude therefrom.
Referring to
Referring to
The camera used may be any camera producing digital images; suitable parameters are as follows: aperture of 8.0 or more, a zoom of 80 mm or more (related to a full-frame sensor), a sensitivity of ISO 800 or more, preferably ISO 1600, automatic adjustment of the shutter speed to the aforementioned settings. Preferably, the camera allows adjusting the angle of the display. Preferably, the camera produces photographs in raw image format. In the present embodiment, for instance, the camera is a Canon PowerShot G1 X Mark II (Canon, Japan).
In the following stages 12a and 12b, (at least) two photo sessions are done, namely, for the auricles of the left ear and right ear, respectively.
Referring to
The photo session of the right ear is done in a like manner as that of the left ear after adapting the position of the additional lighting sources 101-104.
In this step 14, the photos from the three photo sessions are processed to obtain three 3D meshes. The calculation of the 3D meshes can be done using any photogrammetric software allowing to process 2D photos into 3D geometric representations. For instance, in the present embodiment the software Patch-Based Multi-View Stereo (PMVS, https://www.di.ens.fr/pmvs) is used. The calculations are done separately for the left ear, the right ear, and the head, and result in three 3D meshes.
The meshes as directly obtained from these calculations may contain holes or other defects, and the meshes are cleaned up as in a typical post-scan clean-up procedure, which may include: closing small holes in the surface, remeshing to obtain a smooth surface, and/or removing isolated vertices and faces to obtain a single object. Furthermore, in the head mesh, all elements that correspond to the torso and neck are removed and the opening which corresponds to the neck is closed. If desired, the meshes may also be decimated to ease the processing in any of these operations.
The head mesh is scaled corresponding to the head width given by the aforementioned distance between the left and right temples. The auricle meshes are scaled to the respective sizes of the shapes of the ear portions in the head mesh; alternatively, the auricle meshes may be scaled using a distance measured between two or more specific points of the auricle as mentioned above.
The head mesh is smoothed and remeshed to a given number of elements. The inventors found that a number of between 60000 and 80000 is preferable, corresponding to an average edge length of 2 mm or less. The auricle meshes are smoothed and remeshed as well, preferably to an average edge length of 1 mm, which typically will yield a number of elements between 5000 and 6000.
The three 3D meshes are merged to a joint 3D object, referred to as combined mesh—step 15 of
This step 16 is optional, but was found to be advantageous in many cases. The combined mesh is then adapted by rotation and translation as needed to align it with a desired orientation as illustrated in
In this step 17, HRTF calculations are done based on the combined mesh, by means of boundary element methods (BEM) known in the art. For this any suitable software may be used, such as Mesh2HRTF (http://mesh2hrtf.sourceforge.net). HRTFs are calculated for both ears and various sound sources. At least 1000 source positions are used for a single distance. The calculations are done for at least 50 frequencies in the range between 100 Hz and 20 kHz. When used with Mesh2HRTF, a SOFA file describing all HRTFs of a listener is created. The calculations are done on a suitable computer system 2 (
Optionally, the quality of the HRTFs may be evaluated—step 18 of
The evaluation step may, for instance, comprise two parts that check the calculated HRTF set on a perceptually relevant basis for its spatial entropy and compatibility to HRTF sets of a database. For these checks, established computational auditory models of spatial perception are used that require binaural stimuli and reference HRTFs as input. One example for a suitable model is the sagittal-plane sound localisation model, as explained in: Baumgartner, Majdak, and Laback, “Modeling sound-source localization in sagittal planes for human listeners,” J. Acoust. Soc. Am. (2014) 136, 791-802; other models may be suitable as well. The spatial entropy is assessed by using the calculated HRTF as both the binaural stimuli and the reference HRTFs. It thus evaluates the perceptual spatial quality of the calculated HRTFs for a listener being fully adapted to the calculated HRTFs (best case). The compatibility is assessed by using the calculated HRTFs only for the stimulus; for the reference HRTFs, other HRTFs are drawn from a database; databases suitable for this purpose are available in state of the art including the internet (e.g., https://www.sofaconventions.org/mediawiki/index.php/Files). This evaluates the perceptual spatial quality of the calculated HRTFs for a listener not being adapted to the calculated HRTFs (worst case).
For example, this evaluation routine may be realized by using the sound localisation model of Baumgartner, Majdak, and Laback (op. cit.). This model predicts measures of human sound localisation performance, such as the quadrant error rate and local polar RMS error, based on binaural stimuli and a set of reference HRTFs.
In the case the result from step 18 indicates that the HRTFs are not satisfactory, the calculation and evaluation steps 17 and 18, if necessary also the adapting step 16 before them, are repeated until sufficiently accurate HRTFs have been obtained. For this, further modifications of the auricle meshes are made in this step 19. Such a modification will usually serve to compensate possible deviations of the meshes from the shapes of components as shown in the photographs (visual representation). For instance, a typical modification affects the representations of the concha, the navicular fossa, and/or the scapha, which may be shallow as compared to the actual shape of the respective ear component(s); these parts of the mesh are then improved by hollowing out, creating more deep concha, navicular fossa, and/or scapha, which also results in a “sharper” antihelix and antitragus (i.e., having stronger curvatures). Once the modifications are introduced, at least the calculation and evaluation steps are performed again. This is repeated until a pre-defined criterion is met. For instance, the criterion may be that certain quantities fall below (or rise above) predefined thresholds, where suitable thresholds can be derived from the HRTFs of the database used.
In the example where the sound localisation model is used to predict the quadrant error rate and local polar RMS error, the criterion is that (i) the spatial entropy is high enough to obtain predicted quadrant error rates below 18% and local polar RMS errors below 35° and (ii) the compatibility check yields mean quadrant error rates below 30% and local polar RMS errors below 44°.
In the case that the criterion is met, the HRTFs are stored to a data storage medium 3 as described under step D.
| Number | Date | Country | Kind |
|---|---|---|---|
| 18162509.6 | Mar 2018 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2019/056683 | 3/18/2019 | WO | 00 |
