Patent Grant
8595009
The present invention relates generally to audio signal processing. More specifically, embodiments of the present invention relate to methods and apparatuses for performing song detection on audio signals.
In many audio applications, audio signals are recorded. For example, in a frequency modulation (FM) recording application in mobile phones, tablet computers, or other portable devices, FM programs can be recorded in response to user operations on recording buttons or based on a reservation. Recorded audio signals may include a mixture of song, speech (including speech-over-music), noise, silence, etc. Users may desire to only save individual songs in the recorded audio signals.
An approach has been proposed to detect songs from audio signals based on repeating occurrences of audio segments in the audio signals, assuming that a repeated long audio segment is a song while speech seldom repeats for multiple times. An example implementation of the approach can be found in PopCatcher Internet Radio Recorder Application from PopCatcher AB, Hastholmsvagen 28, 5tr, 131 40 Nacka, SWEDEN, which is herein incorporated by reference for all purposes.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated.
According to an embodiment of the invention, a method of performing song detection on an audio signal is provided. Clips of the audio signal are classified into classes comprising music. Class boundaries of the music clips are detected as candidate boundaries. At least one combination including one or more non-overlapped sections bounded by the candidate boundaries are derived. Each of the sections meets the following conditions: 1) including at least one music segment longer than a predetermined minimum song duration as a candidate song, 2) shorter than a predetermined maximum song duration, 3) both starting and ending with a music clip, and 4) a proportion of the music clips in each of the sections is greater than a predetermined minimum proportion.
According to another embodiment of the invention, an apparatus for performing song detection on an audio signal is provided. The apparatus includes a classifying unit, a boundary detector and a song searcher. The classifying unit classifies clips of the audio signal into classes comprising music. The boundary detector detects class boundaries of the music clips as candidate boundaries. The song searcher derives at least one combination including one or more non-overlapped sections bounded by the candidate boundaries. Each of the sections meets the following conditions: 1) including at least one music segment longer than a predetermined minimum song duration as a candidate song, 2) shorter than a predetermined maximum song duration, 3) both starting and ending with a music clip, and 4) a proportion of the music clips in each of the sections is greater than a predetermined minimum proportion.
Further features and advantages of the invention, as well as the structure and operation of various embodiments of the invention, are described in detail below with reference to the accompanying drawings. It is noted that the invention is not limited to the specific embodiments described herein. Such embodiments are presented herein for illustrative purposes only.
Additional embodiments will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.
The present invention is illustrated by way of examples, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:
The embodiments of the present invention are below described by referring to the drawings. It is to be noted that, for purpose of clarity, representations and descriptions about those components and processes known by those skilled in the art but unrelated to the present invention are omitted in the drawings and the description.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system (e.g., an online digital media store, cloud computing service, streaming media service, telecommunication network, or the like), device (e.g., a cellular telephone, portable media player, personal computer, television set-top box, or digital video recorder, or any media player), method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof.
A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
As illustrated in
Audio signal 110 to be processed by apparatus 100 includes a plurality of consecutive clips. Each clip includes a plurality of consecutive frames. The length of the clips and the length of the frames depend on the requirement of the classification model for classifying the clips.
Classification
Classifying unit 101 classifies the clips of audio signal 110 into classes comprising music. In the context of this specification, the term “music” includes songs with instrumental sound and songs without instrumental sound.
The classification model may be trained based on training sample sets for the classes to be identified (e.g., music). Various models for classifying objects may be adopted. For example, the classification model may be based on adaBoost, Support Vector Machine, Hidden Markov Model, or Gaussian Mixture Model.
Various features for characterizing the difference between audio signals of the classes to be identified may be adopted in the classification model. For example, the features of each frame (also called as frame-level features) may comprise at least one of timbre-related feature and chroma feature. The timbre-related feature may be used to distinguish different types of sound production such as music, speech, etc. For example, the timbre-related feature may comprise at least one of zero-crossing rate, short-time energy, sub-band spectral distribution, spectral flux and Mel-frequency Cepstral Coefficient. Chroma feature may be used to represent the melody information of an audio signal. For example, chroma feature is generally defined as a 12-dimensional vector where each dimension corresponds to the intensity of a semitone class (there are 12 semitones in an octave).
In an example implementation of classifying unit 101, classifying unit 101 may calculate frame-level features of frames in each clip and derive features for characterizing variation of the frame-level features (also called as clip-level features) from the frame-level features of the clip. The clip-level features may be used to capture the rhythmic property of different sounds and especially to differentiate speech and music. For example, the clip-level features of a clip may comprise mean and standard deviation of the frame-level features of the clip, and/or rhythmic feature. The rhythmic feature of a clip may be used to capture regular recurrence or pattern in the frame-level features of the clip. For example, the rhythmic feature comprises at least one of rhythm strength, rhythm regularity, rhythm clarity and two dimension (2D) sub-band modulation. Each clip may be classified based on the corresponding clip-level features.
The function of calculating the features may be implemented in classifying unit 101, or may be implemented in a separate feature extractor (not illustrated in
In some circumstances, song signals recorded in audio signal 110 may include noise due to short time interference or other factors. In a further embodiment of classifying unit 101, the classes identified by classifying unit 101 may further comprise noise. Classifying unit 101 may further re-classify any noise segment adjoining with two music clips and having a length smaller than a threshold as music. The threshold may be obtained based on statistics on length of noise in sample song recordings. In this way, true song signal which incorrectly recorded as noise can be corrected as music class.
In some circumstances, clips in songs may be incorrectly classified as non-music. The clips generally present as sudden changes in long music segments. In a further embodiment of classifying unit 101, classifying unit 101 may further calculate confidence for the class of each of the clips. Classifying unit 101 may comprise a first median filter and one or more second median filters with different smoothing windows. The first median filter smoothes the clips from the start to the stop of the audio signal. For each current clip, if the confidence of the clip is lower than a threshold and the class of the clip is different from the median of the classes of the clips in a smoothing window centered at the clip, the class of the clip is updated with the median. The threshold is used to determine whether a confidence can indicate a correct classification. It can be set in advance, or can be learned by testing the classifier with a sample set. The second median filters with different smoothing windows smooth the clips subsequently. In this way, such incorrectly classified clips can be reclassified as music.
Detecting Candidate Boundaries
A—Detecting Based on Classification
Because every song can exhibit as a segment of one or more consecutive music clips (also called as music segment in the following), class information of the clips in audio signal 110 may reveal one kind of information on true songs included in audio signal 110. Specifically, every music segment may be found from audio signal 110 based on the class information of the clips, and the music segment may be viewed as estimation to the corresponding true song.
Boundary detector 102 detects class boundaries of the music clips (between music clip and non-music clip) as candidate boundaries 120. In this way, music segments which may be estimated as true songs can be detected.
B—Detecting Based on Feature Dissimilarity
Further, in case of continuous playing, for example, two or more consecutive songs can also exhibit as one music segment (e.g., music mixing or sampling). In this case, a sole music segment determined according to the class information is not always sufficient to discover the true boundary of the songs. It is possible to improve this estimation by exploiting the fact that for two segments belonging to different songs, features of signals in the different segments may exhibit some different characteristics (that is, lower consistency/higher dissimilarity).
In a further embodiment of boundary detector 102, boundary detector 102 may also detect positions as candidate boundaries 120 if feature dissimilarities between two windows disposed about the position within any music segment in audio signal 110 is higher than a threshold THD. The threshold THD may be determined based on statistics on feature dissimilarities calculated from sample signals including consecutive songs. In this way, it is possible to detect candidate boundaries for separating consecutive songs. To distinguish the candidate boundaries detected based on classification and based on feature dissimilarity, the candidate boundaries detected based on classification are called as a first type and the candidate boundaries based on feature dissimilarity are called as a second type.
Various methods of evaluating the feature dissimilarity between features of two windows can be adopted in boundary detector 102. For example, the feature dissimilarity between two windows may be calculated as Kullback-Leibler Divergence (KLD).
In an example, the feature dissimilarity DsKLD may be calculated as a symmetric KLD by
where Cl and Cr are covariance matrices of features extracted from frames of the left window and the right window respectively, ul and ur are corresponding means, tr[X] is the sum of diagonal elements of a matrix X.
Various features extracted from frames may be used for calculating the feature dissimilarity. The function for calculating the features may be included in boundary detector 102, or may be implemented in a separate feature extractor (not illustrated in
In an example, the threshold THD is determined as an adaptive threshold thseg(α)
thseg(α)=mean+α·std (2)
where mean and std is the mean and standard deviation of the calculated feature dissimilarity respectively, and α is a tuning parameter, typically in a range from 0 to about 3 (e.g., equal to 1.2).
C—Verifying Based on Content Coherence
In audio signal 110, the candidate boundaries may be boundaries of true songs. It is possible to judge whether the candidate boundaries are boundaries of true songs or not by investigating a broad range (if compared with the windows for calculating the feature dissimilarity in candidate boundary detector) of segments surrounding the candidate boundaries. The content coherence (distance) serves as a metric to further jud0ge if a candidate boundary is a true song start/stop boundary. If the content coherence (distance) is large (small), the content of the surrounding segments is similar and thus the candidate boundary is not a true song start/stop boundary; otherwise, if the content coherence (distance) is small (large), the boundary is true.
In a further embodiment of boundary detector 102, for each boundary t of the candidate boundaries, boundary detector 102 calculates at least one content coherence distance between two windows (e.g., one minute long) surrounding the boundary t. If more than one content coherence distances are calculated for one boundary, features for calculating the content coherence distances are at least partly different from each other.
Various methods of calculating coherent distance between two contents may be adopted.
Various features may be adopted to calculate the content coherence distance. For example, features for calculating the content coherence distance may comprise at least one of chroma feature, timbre-related feature and Rhythm-related feature. In a further example, the Rhythm-related feature may be obtained through at least one of tempo estimation, beat/bar detection and rhythm pattern extraction.
For each boundary t of the candidate boundaries, boundary detector 102 calculates a possibility (e.g., confidence) that boundary t is the true boundary of a song based on the at least one corresponding content coherence distance. Various methods may be adopted to calculate the possibility. For example, a sigmoid function may be adopted to calculate the possibility. For another example, the possibility conf may be calculated based on the content coherence distance Dcoh as
where Thlb and Thub are the lower-bound threshold and upper-bound threshold respectively, VH (e.g., 1) is a value representing that boundary t is true, VM (e.g., 0) is a value representing that boundary t is false, and VM (e.g., 0.5) is a value representing that boundary t is uncertain yet (neither true nor false).
If multiple content coherence distances are computed based on different features, they can be combined in various ways. For example, it is possible to set the possibility to VH if all the content coherence distances are larger than the corresponding upper-bound thresholds, or more loosely, if any one of the content coherence distances is larger than the corresponding upper-bound threshold. Another probabilistic way is to build a model to represent the joint distribution model of these distances based on a training set.
If the possibility indicates that boundary t is a false boundary, boundary detector 102 may perform the following processing.
If boundary t is within a music segment, boundary detector 102 may remove boundary t if the music segment including only boundary t and bounded by two candidate boundaries has a length smaller than the predetermined maximum song duration.
If a speech segment bounded by boundary t and another candidate boundary has a length smaller than a threshold, boundary detector 102 may identify the two candidate boundaries as to-be-removed. The threshold may be obtained based on statistics on speech segments between two songs.
Boundary detector 102 may remove all the to-be-removed candidate boundaries, or boundary detector 102 may change one or more pairs of two to-be-removed candidate boundaries bounding a music segment as the second type and remove the remaining to-be-removed candidate boundaries.
In a further embodiment of boundary detector 102, in case that the possibility neither indicates that boundary t is a true boundary nor indicates that boundary t is a false boundary, if boundary t is of the second type (that is, within a music segment), boundary detector 102 may calculate a probability P(H0) that two music segments of durations l1 and l2 adjoining with each other at boundary t are two true songs with a pre-trained song duration model, and calculate a probability P(H1) that a music segment obtained by merging the two music segments is a true song with the pre-trained song duration model. If the following condition is not met, boundary detector 102 remove boundary t:
wherein the pre-trained song duration model is a Gaussian model G(l;μ,σ).
D—Verifying Based on Repetitive Sections
In a further embodiment of boundary detector 102, boundary detector 102 may search for one or more pairs of two repetitive sections [t1, t2] and [t1+l, t2+l] in audio signal 110, where the lag l is shorter than the predetermined maximum song duration.
In general, in comparison with other kinds of content, songs may exhibit unique characteristics by including repetitive sections, i.e., segments with the same melody. It is possible to assume a section [t1, t2+l] between the repetitive sections [t1, t2] and [t1+l, t2+l] as belonging to one song. Therefore, if one candidate boundary in the section [t1, t2+l] is within a music segment, boundary detector 102 may remove the candidate boundary. If a speech segment in the section [t1, t2+l] bounded by two candidate boundaries has a length smaller than a threshold, boundary detector 102 may identify the two candidate boundaries as to-be-removed. Boundary detector 102 may remove all the to-be-removed candidate boundaries, or may change one or more pairs of two to-be-removed candidate boundaries bounding a music segment as the second type and remove the remaining to-be-removed candidate boundaries. The threshold may be obtained based on statistics on the length of music segments misclassified as speech in sample songs.
In this way, candidate boundaries may be verified based on repetitive sections in the audio signal, reducing the possibility that false boundaries between songs are detected as true song boundaries.
Various methods of detecting repetitive sections in audio signals may be adopted by boundary detector 102 to search for repetitive sections in the segments. For example, methods based on similarity matrix or time-lag similarity matrix may be adopted.
In a further embodiment of boundary detector 102, boundary detector 102 may calculate an adaptive threshold for binarizing the similarity matrix based on a percentile. In case of sorting similarity values in the similarity matrix in descending order, only the first small percentage of the similarity values depending on the percentile are binarized to a value representing repetition. The percentile is a product of the proportion of the music clips in the corresponding segment and a pre-defined base percentile. In this way, the percentile and the adaptive threshold are both adaptive to the proportion of music content in the segment.
In a further embodiment of boundary detector 102, boundary detector 102 may only search for the repetitive sections longer than a threshold. The threshold may be obtained based on statistics on the length of repetitive sections in sample songs. In this way, only those repetitive sections long enough can be detected.
In a further embodiment of boundary detector 102, boundary detector 102 may search for sections [t1, t2] and [t1+l, t2+l] such that the music clips are in the majority of section [t1, t2+l]. For example, the proportion of the clips classified as music in section [t1, t2+l] is greater than 50%. For another example, the proportion ml of the clips classified as music in the section [t1, t2], the proportion m2 of the clips classified as music in the section [t1+l, t2+l], the proportion mc of the clips classified as music in the section [t2, t1+l] and the sum ms of m1, m2 and mc may meet some conditions, such as one of the following conditions:
m1>0.5 and m2>0.5 and mc>0.5 condition 1:
m1>0.1 and m2>0.1 and mc>0.1 and ms>1.8. condition 2:
In these ways, it is possible to reduce the chance of detecting non-music sections such as speech sections as repetitive sections.
It should be noted that, in case of verifying the candidate boundaries based on both content coherence and repetitive sections, they can be performed in either order.
In a further embodiment of boundary detector 102, boundary detector 102 may merge two of the candidate boundaries spaced with a distance smaller than a threshold as one candidate boundary. The threshold may be a value smaller than or equal to the minimum song duration. The merged candidate boundary may be any one position between the two candidate boundaries.
Song Detection
Returning to
1) including at least one music segment longer than a predetermined minimum song duration (called as candidate song),
2) shorter than a predetermined maximum song duration,
3) both starting and ending with a music clip, and
4) a proportion of the music clips in each of the sections is greater than a predetermined minimum proportion.
The predetermined minimum song duration and the predetermined maximum song duration may be determined from statistics on length of various songs, or may be specified by a user who desires songs of a length within a specific range.
Any portion bounded between two candidate boundaries in the audio signal meeting conditions 1) to 4) may be regarded as a possible section. Therefore, there may be multiple possible sections in the audio signal. The possible sections not overlapped with each other may be selected to form a combination. Alternatively, depending on specific application requirements, the number of sections in combinations may be set to a specific number, e.g., 2, 3 and so on.
In this way, various possible song partitions in the audio signal may be obtained as the derived combinations. Based on these combinations, a desired song partition may be selected manually or automatically.
Two candidate boundaries bounding a possible section may be subsequent, that is to say, there is no other candidate boundary between the two candidate boundaries. In this case, the possible section is an undividable music segment. For example, Candidate boundaries b and c bounds an undividable music segment [b, c]. Two candidate boundaries bounding a possible section may also include one or more other candidate boundaries. In this case, the possible section includes at least two undividable segments. For example, possible section [a, c] includes two undividable segments [a, b] and [b, c], and possible section [b, e] includes undividable segments [b, c], [c, d] and [d, e].
In case of forming a combination including only one section, any possible section may be selected. In case of a combination including more than one section, at least two possible sections which are not overlapped with each other may be selected as sections to form a combination. Different combinations may have a different number of sections. For example, from the audio signal in
If the possibility based on the content coherence distance indicates that a candidate boundary is true, this candidate boundary cannot be within any section of the combinations. In a further embodiment of song searcher 103, in deriving a combination, song searcher 103 excludes any combination including a section where the possibility corresponding to one candidate boundary within the section indicates that the candidate boundary is a true boundary. That is to say, the possibility corresponding to each candidate boundary within the sections does not indicate that the candidate boundary is a true boundary.
In a further embodiment of song searcher 103, song searcher 103 may detect each music segment bounded by two subsequent candidate boundaries t1 and t2 and longer than the predetermined minimum song duration as a candidate song, and form the combination by including the candidate song [t1, t2] or their extensions as a section. The sections in the formed combination are not overlapped with each other, and also meet the above-mentioned conditions 1) to 4). Each extension may be obtained by at least one of the followings:
extending the boundary t1 of the candidate song [t1, t2] to the candidate boundary t1−l1 of a music segment [t1−l1, t1−l2] in the left direction; and
extending the boundary t2 of the candidate song [t1, t2] to the candidate boundary t2+l4 of a music segment [t2+l3, t2+l4] in the right direction.
In this way, the case where some impossible combinations are obtained and then are excluded by verifying whether they meet the conditions is likely to be avoided, thus reducing the computation cost.
In case that boundary detector 102 verifies the candidate boundaries based on content coherence as described in the above, in a further embodiment of song searcher 103, song searcher 103 may obtains the extensions in a way such that:
the extending in the left direction is stopped if the possibility based on content coherence distance of the candidate boundary t1−l1 of the music segment [t1−l1, t1−l2] being extended to indicates that the candidate boundary t1−l1 is a true song boundary, and
the extending in the right direction is stopped if the possibility based on content coherence distance of the candidate boundary t2+l4 of the music segment [t2+l3, t2+l4] being extended to indicates that the candidate boundary t2+l4 is a true song boundary.
In this way, it is possible to exclude the sections including a true song boundary, thus improving the accuracy of the song detection.
Further, it is possible to incorporate a requirement that if a non-music (e.g., speech) segment is to be included in performing the extending and the non-music segment is longer than a pre-defined threshold, the extending may be stopped.
In a further embodiment of song searcher 103, more than one combination may be derived by song searcher 103. In this case, song searcher may further separate the combinations into different groups. Every combination in each group includes the same candidate song(s) and each section in the combination includes the same candidate song(s) with one section in another combination of the same group. In the example illustrated in
As illustrated in
In an example implementation of step 503, it is possible to calculate frame-level features of frames in each clip and derive clip-level features for characterizing variation of the frame-level features from the frame-level features of the clip. The clip-level features may be used to capture the rhythmic property of different sounds and especially to differentiate speech and music.
In a further implementation of step 503, the classes identified at step 503 may further comprise noise. It is possible to further re-classify any noise segment adjoining with two music clips and having a length smaller than a threshold as music. The threshold may be obtained based on statistics on length of noise in sample song recordings.
In a further implementation of step 503, it is possible to further calculate confidence for the class of each of the clips. Further, it is possible to smooth the clips from the start to the stop of the audio signal with a smoothing window. For each current clip, if the confidence of the clip is lower than a threshold and the class of the clip is different from the median of the classes of the clips in the smoothing window centered at the clip, the class of the clip is updated with the median. Further, it is possible to smooth the clips with different smoothing windows. The threshold is used to determine whether a confidence can indicate a correct classification. It can be set in advance, or can be learned by testing the classifier with a sample set.
At step 505, class boundaries of the music clips are detected as candidate boundaries.
In a further implementation of step 505, it is also possible to detect positions as candidate boundaries if feature dissimilarities between two windows disposed about the position within any music segment in the audio signal is higher than the threshold THD.
Various methods of evaluating the feature dissimilarity between features of two windows can be adopted at step 505. For example, the feature dissimilarity between two windows may be calculated as Kullback-Leibler Divergence (KLD).
In an example, the feature dissimilarity DsKLD may be calculated as a symmetric KLD by Eq. (1). Various features extracted from frames may be used for calculating the feature dissimilarity.
In a further implementation of step 505, for each boundary t of the candidate boundaries, it is possible to calculate at least one content coherence distance between two windows (e.g., one minute long) surrounding the boundary t. If more than one content coherence distances are calculated for one boundary, features for calculating the content coherence distances are at least partly different from each other.
For each boundary t of the candidate boundaries, a possibility (e.g., confidence) that boundary t is the true boundary of a song is calculated based on the at least one corresponding content coherence distance. Various methods may be adopted to calculate the possibility. For example, a sigmoid function may be adopted to calculate the possibility. For another example, the possibility conf may be calculated based on the content coherence distance Dcoh by Eq. (3).
If multiple content coherence distances are computed based on different features, they can be combined in various ways. For example, it is possible to set the possibility to VH if all the content coherence distances are larger than the corresponding upper-bound thresholds, or more loosely, if any one of the content coherence distances is larger than the corresponding upper-bound threshold. Another probabilistic way is to build a model to represent the joint distribution model of these distances based on a training set.
If the possibility indicates that boundary t is a false boundary, it is possible to perform the following processing.
If boundary t is within a music segment, boundary t may be removed if the music segment including only boundary t and bounded by two candidate boundaries has a length smaller than the predetermined maximum song duration.
If a speech segment bounded by boundary t and another candidate boundary has a length smaller than a threshold, the two candidate boundaries may be identified as to-be-removed. The threshold may be obtained based on statistics on speech segments between two songs.
All the to-be-removed candidate boundaries may be removed, or one or more pairs of two to-be-removed candidate boundaries bounding a music segment may be changed as the second type and the remaining to-be-removed candidate boundaries may be removed.
In a further implementation of step 505, in case that the possibility neither indicates that boundary t is a true boundary nor indicates that boundary t is a false boundary, if boundary t is of the second type (that is, within a music segment), a probability P(H0) that two music segments of durations l1 and l2 adjoining with each other at boundary t are two true songs may be calculated with a pre-trained song duration model, and a probability P(H1) that a music segment obtained by merging the two music segments is a true song may be calculated with the pre-trained song duration model. If the condition defined by Eq. (4) is not met, it is possible to remove boundary t.
In a further implementation of step 505, it is possible to search for one or more pairs of two repetitive sections [t1, t2] and [t1+l, t2+l] in the audio signal, where the lag l is shorter than the predetermined maximum song duration.
If one candidate boundary in the section [t1, t2+l] is within a music segment, it is possible to remove the candidate boundary. If a speech segment in the section [t1, t2+l] bounded by two candidate boundaries has a length smaller than a threshold, it is possible to identify the two candidate boundaries as to-be-removed. All the to-be-removed candidate boundaries may be removed, or one or more pairs of two to-be-removed candidate boundaries bounding a music segment may be changed as the second type and the remaining to-be-removed candidate boundaries may be removed. The threshold may be obtained based on statistics on the length of music segments misclassified as speech in sample songs.
Various methods of detecting repetitive sections in audio signals may be adopted to search for repetitive sections in the segments. For example, methods based on similarity matrix or time-lag similarity matrix may be adopted.
In a further implementation of step 505, it is possible to calculate an adaptive threshold for binarizing the similarity matrix based on a percentile. In case of sorting similarity values in the similarity matrix in descending order, only the first small percentage of the similarity values depending on the percentile are binarized to repetition. The percentile is a product of the proportion of the music clips in the corresponding segment and a pre-defined base percentile.
In a further implementation of step 505, it is possible to only search for the repetitive sections longer than a threshold. The threshold may be obtained based on statistics on the length of repetitive sections in sample songs.
In a further implementation of step 505, it is possible to search for sections [t1, t2] and [t1+l, t2+l] such that the music clips are in the majority of section [t1, t2+l]. For example, the proportion of the clips classified as music in section [t1, t2+l] is greater than 50%. For another example, the proportion m1 of the clips classified as music in the section [t1, t2], the proportion m2 of the clips classified as music in the section [t1+l, t2+l], the proportion mc of the clips classified as music in the section [t2, t1+l] and the sum ms of m1, m2 and mc may meet some conditions, such as one of the following conditions:
m1>0.5 and m2>0.5 and mc>0.5 condition1:
m1>0.1 and m2>0.1 and mc>0.1 and ms>1.8. condition 2:
It should be noted that, in case of verifying the candidate boundaries based on both content coherence and repetitive sections, they can be performed in either order.
In a further implementation of step 505, it is possible to merge two of the candidate boundaries spaced with a distance smaller than a threshold as one candidate boundary. The threshold may be a value smaller than or equal to the minimum song duration. The merged candidate boundary may be any one position between the two candidate boundaries.
At step 507, at least one combination including non-overlapped sections bounded by the candidate boundaries is derived. The sections meet the above conditions 1) to 4).
The predetermined minimum song duration and the predetermined maximum song duration may be determined from statistics on length of various songs, or may be specified by a user who desires songs of a length within a specific range.
Any portion bounded between two candidate boundaries in the audio signal meeting conditions 1) to 4) may be regarded as a possible section. Therefore, there may be multiple possible sections in the audio signal. The possible sections not overlapped with each other may be selected to for a combination. Alternatively, depending on specific application requirements, the number of sections in combinations may be set to a specific number, e.g., 2, 3 and so on.
In a further implementation of step 507, it is possible to detect each music segment bounded by two subsequent candidate boundaries t1 and t2 and longer than the predetermined minimum song duration as a candidate song, and form the combination by including the candidate song [t1, t2] or their extensions as a section. The sections in the formed combination are not overlapped with each other, and also meet the above-mentioned conditions 1) to 4). Each extension may be obtained by at least one of the followings:
extending the boundary t1 of the candidate song [t1, t2] to the candidate boundary t1−l1 of a music segment [t1−l1, t1−l2] in the left direction; and
extending the boundary t2 of the candidate song [t1, t2] to the candidate boundary t2+l4 of a music segment [t2+l3, t2+l4] in the right direction.
In case of verifying the candidate boundaries based on content coherence as described in the above, in a further implementation of step 507, it is possible to obtain the extensions in a way such that:
the extending in the left direction is stopped if the possibility based on the content coherence distance of the candidate boundary t1−l1 of the music segment [t1−l1, t1−l2] being extended to indicates that the candidate boundary t1−l1 is a true son boundary, and
the extending in the right direction is stopped if the possibility based on the content coherence distance of the candidate boundary t2+l4 of the music segment [t2+l3, t2+l4] being extended to indicates that the candidate boundary t2+l4 is a true song boundary.
Further, it is possible to incorporate a requirement that if a non-music (e.g., speech) segment is to be included in performing the extending and the non-music segment is longer than a pre-defined threshold, the extending may be stopped.
Method 500 ends at step 509.
In a further implementation of step 507, more than one combination may be derived. In this case, step 507 may further comprise separating the combinations into different groups. Every combination in each group includes the same candidate song(s) and each section in the combination includes the same candidate song(s) with one section in another combination of the same group. For every two combinations of different groups, at least one section in one of the two combinations does not include the same candidate song(s) with each section in another of the two combinations.
As illustrated in
For each combination, song evaluator 604 evaluates a possibility that all the intervals for separating the sections represent true song partitions with an evaluation model trained based on at least one of song duration, interval between songs, and song probability.
Some characteristics are observed that, for two subsequent songs, duration of the songs complies with a song duration distribution, and non-song duration (interval) between the songs complies with a song interval distribution. Further, features extracted from the songs exhibit some characteristics different from that of non-songs.
For each combination, every section in the combination is assumed as a true song, and the combination represents a possible song partition in the audio signal. One or more of the above characteristics may be adopted to determine whether the combination can represent a true song partition. For example, it is possible to train a song duration model for evaluating whether a section is a true song based on statistics on durations of a set of sample songs, and estimate the possibility that a section is a true song with the trained model based on the length of the section. For another example, it is possible to train a non-song model for evaluating whether the portion between two adjacent sections is a non-song based on statistics on intervals between subsequent sample songs, and estimate the possibility that the portion between two subsequent sections is non-song with the trained model based on the interval between the sections. For another example, it is possible to train a song probability model for evaluating whether a section is a true song based on the features extracted from a set of sample songs, and estimate the possibility that a section is a true song with the trained model based on the features extracted from the section. Other criteria may also be adopted to determine whether the combination can represent a true song partition. If more than one possibility is obtained, it is possible to combine them in a joint model to obtain a final possibility. For example, it is possible to calculate mean or a joint probability function of respective possibilities.
In an example of the joint probability function, the final possibility may be calculated in form of average or product of confidence P([e, s]) for all the intervals [e, s] for separating the one or more sections in the corresponding combination, where if one intervals [e, s] separates two adjacent sections [s1,e] and [s,e2], the confidence P([e, s]) is calculated as
P([e,s])=Pdur([s1,e])Pdur([s,e2])αPnsβ([e,s])Psong([s1,e])Psong([s,e2]) (5-1) and
if there is only one section [x,y] in the corresponding combination, the confidence P([e, s]) is calculated as
P([e,s])=Pdur([x,y]) Psong([x,y]) (5-2)
where Pdur( ) (is a pre-trained song duration model, Pns( )is a pre-trained non-song duration model which is estimated as a Gamma distribution, Psong( ) is a song probability model indicating the probability that a section is a true song, and α and β are flatting coefficients to deal with the different scales of different probabilistic distributions.
Selector 605 selects one combination with the highest possibility. Sections in the combination are regarded as true songs.
In a further embodiment of selector 605, for each boundary b of every section in the selected combination, selector 605 may calculate a log likelihood difference ΔBIC(t) based on a Bayesian Information Criteria (BIC) based method for each frame position t in a BIC window centered at boundary b, and adjust boundary b to the frame position t corresponding to a peak ΔBIC(t).
In a further embodiment of selector 605, selector 605 may adjust boundary b to be refined to frame position t corresponding to the peak ΔBIC(t) closer to boundary b than frame position t′ corresponding to another peak ΔBIC(t′).
In an alternative embodiment of selector 605, for each boundary b of every section in the selected combination, selector 605 may calculate a value RΔBIC(t|b)=ΔBIC(t)·Pst(|t−b|) for each frame position t in a BIC window centered at boundary b, where ΔBIC(t) is a log likelihood difference calculated based on a Bayesian Information Criteria (BIC) based method, and Pst( )is a shift time duration model based on a Gaussian distribution with zero mean. Further, selector 605 may adjust boundary b to frame position t corresponding to the highest peak RΔBIC(t).
In an example, the frame-level features may comprise chroma feature.
As illustrated in
At step 809, for each derived combination, a possibility that all the intervals for separating the sections represent true song partitions is calculated with an evaluation model trained based on at least one of song duration, interval between songs, and song probability.
For each derived combination, every section in the combination is assumed as a true song, and the combination represents a possible song partition in the audio signal. One or more of the above characteristics may be adopted to determine whether the combination can represent a true song partition. Other criteria may also be adopted to determine whether the combination can represent a true song partition. If more than one possibility is obtained, it is possible to combine them in a joint model to obtain a final possibility. For example, it is possible to calculate mean or a joint probability function of respective possibilities.
In an example of the joint probability function, the final possibility may be calculated in form of average or product of confidence P([e, s]) for all the intervals [e, s] for separating the one or more sections in the corresponding combination based on Eqs. (5-1) and (5-2).
At step 811, one combination with the highest possibility is selected. Sections in the combination are regarded as true songs.
In a further implementation of step 811, for each boundary b of every section in the selected combination, it is possible to calculate a log likelihood difference ΔBIC(t) based on a Bayesian Information Criteria (BIC) based method for each frame position t in a BIC window centered at boundary b, and adjust boundary b to the frame position t corresponding to a peak ΔBIC(t).
In a further implementation of step 811, it is possible to adjust boundary b to be refined to frame position t corresponding to the peak ΔBIC(t) closer to boundary b than frame position t′ corresponding to another peak ΔBIC(t′).
In an alternative implementation of step 811, for each boundary b of every section in the selected combination, it is possible to calculate a value RΔBIC(t|b)=ΔBIC(t)·Pst(|t−b|) for each frame position t in a BIC window centered at boundary b, where ΔBIC(t) is a log likelihood difference calculated based on a Bayesian Information Criteria (BIC) based method, and Pst( )is a shift time duration model based on a Gaussian distribution with zero mean. Further, it is possible to adjust boundary b to frame position t corresponding to the highest peak RΔBIC(t).
In an example, the frame-level features may comprise chroma feature.
In
The CPU 901, the ROM 902 and the RAM 903 are connected to one another via a bus 904. An input/output interface 905 is also connected to the bus 904.
The following components are connected to the input/output interface 905: an input section 906 including a keyboard, a mouse, or the like ; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), or the like, and a loudspeaker or the like; the storage section 908 including a hard disk or the like ; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs a communication process via the network such as the internet.
A drive 910 is also connected to the input/output interface 905 as required. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is mounted on the drive 910 as required, so that a computer program read therefrom is installed into the storage section 908 as required.
In the case where the above-described steps and processes are implemented by the software, the program that constitutes the software is installed from the network such as the internet or the storage medium such as the removable medium 911.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
The following exemplary embodiments (each an “EE”) are described.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2011 1 0243070 | Aug 2011 | CN | national |
This Application claims the benefit of priority to related, co-pending Chinese Patent Application number 201110243070.6 filed on 19 Aug. 2011 and U.S. Pat. Application No. 61/540,346 filed on 28 Sep. 2011 entitled “Method and Apparatus for Performing Song Detection on Audio Signal” by Lu, Lie et al. hereby incorporated by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5876213 | Matsumoto | Mar 1999 | A |
| 6784354 | Lu et al. | Aug 2004 | B1 |
| 6819863 | Dagtas et al. | Nov 2004 | B2 |
| 7336890 | Lu | Feb 2008 | B2 |
| 20020120456 | Berg | Aug 2002 | A1 |
| 20040170392 | Lu et al. | Sep 2004 | A1 |
| 20040216585 | Lu et al. | Nov 2004 | A1 |
| 20060065106 | Pinxteren | Mar 2006 | A1 |
| 20060085188 | Goodwin | Apr 2006 | A1 |
| 20080052516 | Tachibana et al. | Feb 2008 | A1 |
| 20080147218 | Sugino et al. | Jun 2008 | A1 |
| 20090088877 | Terauchi et al. | Apr 2009 | A1 |
| 20100286989 | Urata et al. | Nov 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 1968043 | Sep 2008 | EP |
| Entry |
|---|
| L. Lu, Stan Li, H, J. Zhang. “Content-based Audio Segmentation Using Support Vector Machines”. Proc. of ICME 2001, pp. 956-959, Tokyo, Japan, 2001. |
| Su et al. “An Integrated Approach to Music Boundary Detection”, National Taiwan University, 2009, pp. 1-6. |
| Lu, L. et al., “Content-Based Audio Classification and Segmentation by Using Support Vector Machines”, Multimedia Systems (8), 482-292, 2003. |
| Wakefield, G.H., “Mathematical Representation of Joint TimeChroma Distributions” SPIE, 1999. |
| Lu, L. et al., “Automatic Mood Detection and Tracking of Music Audio Signals” IEEE Transactions on Audio, Speech and Language Processing, 2006. |
| McKinney, M.F. et al. “Features for Audio and Music Classification” Proc. ISMIR, 2003. |
| Freund, Y. et al., “A Short Introduction to Boosting” Journal of Japanese Society for Artificial Intelligence 14(5): 771-780, 1999. |
| Siegler, M. et al., “Automatic Segmentation, Classification and Clustering of Broadcast News Audio”, Proc. DARPA Speech Recognition, 1997. |
| Goto, M., “A Chorus-Section Detecting Method for Musical Audio Signals”, Proc. Acoustics, Speech, and Signal Processing, 2003. |
| Bartsch, M.A. et al. “To Catch a Chorus: Using Chroma-Based Representations for Audio Thumbnailing”, Proc. WASPAA 2001. |
| Lu, L. et al., “Repeating Pattern Discovery and Structure Analysis from Acoustic Music Data”, Proc. the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval, 2004. |
| Chen, S.S. et al., “Speaker, Environment and Channel Change Detection and Clustering Via the Bayesian Information Criterion” DARPA Broadcast News Transcription Workshop, 1998. |
| Number | Date | Country | |
|---|---|---|---|
| 20130046536 A1 | Feb 2013 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61540346 | Sep 2011 | US |
