Patent Grant
7319958
This invention relates generally to acoustic phones and more particularly to networks of multiple phones.
Speech recognition techniques are known in the art. Many such techniques utilize models that are selected as a function of a polyphone network. For example, a network of phones (wherein a “phone” is generally understood to typically comprise a speech sound considered as a physical event without regard to its possible phonological status in the sound system of a language) that comprise at least the primary phonemes (wherein a “phoneme” is generally understood to comprise an abstract unit of the phonological system of a language that corresponds to a set of similar speech sounds that are perceived to be a single distinctive sound in the language) of a given spoken language (or specific dialect thereof can comprise an acoustic foundation upon which one derives a set of models that a corresponding speech recognition engine can utilize to effect recognition of a sample of uttered speech.
Speech recognition platforms (such as, for example, a cellular telephone) that support recognition of multiple languages are also known. Unfortunately, current multilingual (and/or multi-dialect) automatic speech recognition technologies face a number of practical constraints that appear to significantly limit widespread commercial application of such an approach. One problem involves the cost of acquiring and/or the relative availability of relevant language resources and expertise. For example, while many hundreds of different acoustic phones are known to be utilized globally in one language or another, any given language (or dialect) tends to use as phonemes only a relatively small subset of this larger pool. The limited availability of resources and technology-savvy expert knowledge required to linguistically characterize a language for speech recognition engineering purposes (including, but not limited to, specifically identifying those phonemes that appropriately characterize a given language) are an impediment to broad language coverage. Further, the time and expense of creating, finding, or otherwise acquiring appropriate acoustic speech data (with associated transcriptions, lexica, and so forth) of acceptable quality and quantity to permit training of speech models often make such endeavors commercially unfeasible, especially for consumer populations that represent a relatively small speaker group and/or an emerging market.
Computational resource limitations present another problem. A not atypical prior art approach combines a plurality of monolingual speech recognition systems to consolidate sufficient capability to support multiple languages and/or dialects. With such an approach, however, requirements for both the necessary language resources and computational resources increase substantially proportionally with each incremental supported language and/or dialect. The costs and/or form factor constraints associated with such needs can again influence designers away from including languages that correspond to smaller speaker populations and/or smaller present marketing opportunities.
To attempt to meet such problems, some effort has been made to consider sharing acoustic models across a plurality of languages and/or dialects. Such an approach typically requires alteration to the fundamental approach by which models are developed (for example, while models that result from the approaches noted earlier tend to preserve substantially all defined phonetic contrasts in a given set of targeted languages or dialects, such models also poorly support any attempt to exploit any cross-language allophonetic coincidences that might also exist; in addition, the set of models is often too large and too finely differentiated for cost-effective and efficient multilingual or multi-dialect automatic speech recognition needs). For example, one approach employs an acoustic feature-based data-driven method to achieve a kind of phone merger. Acoustic models from a collection of monolingual speech recognition systems are compared and acoustically similar models are merged. Such an approach indeed tends to at least reduce to some extent total memory requirements, but such an approach is also relatively indiscriminate with respect to the grouping of acoustically similar phones (for example, this approach tends to readily permit the merger of acoustically similar but phonologically contrastive models). This approach also fails in large measure to supply acoustic models of phones for which little or no data is conveniently available. One suggested alteration to this latter approach has been to constrain the acoustic data-driven method as a function of language knowledge. This approach seeks to retain improved memory requirements while also attempting to address acoustic confusability of distinct phones. Unfortunately, however, this attempt at improvement still remains largely dependent upon the availability and quality of language data.
Another impediment to fielding a commercially useful result is the present practice of tending to represent phone information with a transcription system that favors unique font sets and/or unusual characters that are typically non-alphanumeric, printable characters that are commonly used for special purposes in command line scripting as they have special control interpretations (such as, for example, ‘@’, ‘>’, ‘\’, ‘|’, ‘=’, ‘{’, ‘^’, ‘?’, ‘*’, ‘(’, and so forth) in one or more relevant computer command and control protocols. For example, some characters used by some transcription systems present phone information that is also interpreted by Unix command line interpreters as specific Unix commands. This unfortunate proclivity further complicates the matter of attempting to provide a flexible, efficient polylingual speech recognition method and apparatus.
The above needs are at least partially met through provision of the polyphone network method and apparatus described in the following detailed description, particularly when studied in conjunction with the drawings, wherein:
Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of various embodiments of the present invention. Also, common but well-understood elements that are useful or necessary in a commercially feasible embodiment are typically not depicted in order to facilitate a less obstructed view of these various embodiments of the present invention.
Generally speaking, pursuant to these various embodiments, phonological similarities across a plurality of languages (and/or dialects) are leveraged in a way that is at least partially based upon language-independent factors. More particularly, a plurality of phones that correspond to a plurality of spoken languages can serve to provide a polyphone network as a function, at least in part, of at least one of phonetic similarity as may exist between the phones and at least one language-independent phonological factor. In a preferred embodiment, the plurality of phones correspond to a plurality of languages including at least some languages that are substantially typologically diverse with respect to one another. Such diversity enriches the resultant polyphone network and tends to render such a network more likely to provide satisfactory models even for languages that are not otherwise specifically represented. Pursuant to another preferred approach, at least some of the plurality of phones are based, at least in part, upon at least some phonemes that are themselves substantially unique to only a few (or one) of the represented languages (in one embodiment, all phonemes that are unique to a particular language amongst the plurality of languages are so represented).
In a preferred embodiment, the polyphone network comprises a hierarchical phone correspondence network that preferably organizes merged phone selections in a tree. In one embodiment, the tree can comprise at least one binary branching tree. Depending upon the embodiment, phone information as contained by the polyphone network can be viewed at varying levels of resolution (such that, for example, higher resolution information may include at least some phone information that comprises phonemes for at least some of the plurality of spoken languages and lower resolution information may include phone information that is representative, at least in part, of cross-linguistic and language-independent mutual phonetic similarity and non-contrastiveness of corresponding higher resolution phone information views).
Depending upon the embodiment, the language-independent phonological factor can comprise information such as language-independent phonological contrast, cross-linguistic phone frequency, predetermined linguistic tendencies, and predetermined linguistic universals.
In general, the resultant polyphone network is preferably substantially unbiased towards any particular language, language family, and/or language type.
In a preferred embodiment, the phones are represented in the polyphonic network using a transcription system that comprises only standard ASCII characters (none of which comprises a control character or other special character (hereinafter referred to generally as “special characters”) having a corresponding computer operation command meaning nor to which Unix command line interpreters ascribe special significance) such that the transcription system characters only require a standard font set and substantially avoid programming or scripting collisions.
So configured, a polyphonic network can be provided that will tend to well support polylingual purposes, including at least substantial accommodation of languages that are not specifically represented by the network. The resultant network (and speech recognition models based thereon) can also be well accommodated by relatively modest computational resources including reduced memory needs as compared to prior art counterparts.
Referring now to the drawings, and in particular to
Various transcription systems exist by which to express such phones. It is therefore possible to provide 11 the phones as described above using any of a wide variety of known or hereafter developed transcription systems. As noted earlier, however, most existing transcription systems tend to require either unique fonts and/or include characters that have additional instructional meaning for many processing platforms. In a preferred approach, therefore, the phones are represented using a transcription system that comprises only standard, printable ASCII characters and wherein none of the characters utilized comprise a special character having a corresponding computer operation command meaning with respect to the desired operating environment (for example, when using Unix to implement the relevant platform(s), none of the transcription characters should have a command meaning to a Unix command line interpreters). A description of a new preferred transcription system that accords with these design criteria follows.
This transcription system is an ASCII-encoded speech symbol inventory restricted to lower-case alphabetic characters, numerals, and the non-alphanumeric symbol ‘_’. Each individual symbol is associated with a specific phonetic feature (or feature constellation) dependent on the type of phone represented (consonant or vowel) and its syntactic position in the phone symbol string. In this embodiment, the basic character length of any consonant or vowel phone representation is two; this obligatory symbol pair is referred to as a phone's base sign. Pursuant to a preferred embodiment, a phone's phonetic features are directly encoded into the label. The first symbol of a base sign unambiguously marks the phone as either a consonant or vowel. All non-tonal diacritic symbols are sorted alphabetically behind the base sign and are framed by the marker ‘_’. Tonal diacritics are suffixed to the right diacritic marker of vowel phone strings. An example of a simple consonant phone representation, one with only the two obligatory base-sign positions filled, is ‘kp’, where the first position ‘k’ is associated with the consonant class of phones and indexes ‘velar articulation’ and the consonant second position ‘p’ indexes ‘voiceless plosive’. Thus, ‘kp’ is a voiceless velar plosive (/k/ in the International Phonetic Alphabet “IPA”). A complex vowel symbol string, that is, a vowel representation that includes diacritics suffixed to the base sign, is exemplified by ‘ii_j—4’, where vowel first position ‘i’ indexes ‘unrounded front’, vowel second position ‘i’ signifies ‘close’, vowel third position ‘j’ is ‘palatal offglide’, and forth position ‘4’ is ‘low tone’ (/ij/ in the IPA). The consonant and vowel position classes are defined below, along with an inventory of the position class constituents.
A preferred consonant phone string has the following position structure, from left to right:
The consonant phone symbols and their interpretations are presented by position class in Table 1.
Within a given consonant symbol string, the first and second positions are only occupied by one symbol each. The first position symbol of a consonant phone label comprises a consonant symbol in traditional Romanized orthography (excluding ‘y’ as noted below). This convention unambiguously marks the label as a consonant phone.
The third position is not limited in length, but the diacritic set will preferably be sorted. Note that all symbols in the consonant inventory are lower-case alphabetic characters and that any particular character may be reused with a different meaning in a different syntactic position in the phone string. Thus, a first position ‘p’ means ‘bilabial’ while a second position ‘p’ means ‘voiceless plosive’. The sign ‘pp’, therefore, represents a voiceless bilabial plosive. Secondary places and manners of articulation are given in the third position diacritic class. As an example, an aspirated voiceless bilabial plosive is ‘pp_c_’.
A preferred vowel phone string has the following position structure, from left to right:
The preferred constituents of the vowel position classes are presented in Table 2.
A vowel phone label preferably begins with either a traditional vowel symbol from Romanized orthography or the character ‘y’. Thus, any label beginning with ‘a’, ‘e’, ‘i’, ‘o’, ‘u’, or ‘y’ unambiguously marks a vowel phone.
Like the consonant position-class inventories, within a given phone symbol string each constituent of the first and second obligatory positions are in a disjunctive relation to other constituents in their same class. Furthermore, each constituent of these positions is a lower-case alphabetic character and any particular character may be reused in a different position where it is associated with a distinct meaning. Thus, an unrounded open back vowel is represented by the string ‘aa’, where ‘a’ indexes ‘back unrounded’ in vowel first position and ‘open’ in vowel second position. Non-tonal diacritics are sorted and framed by the diacritic marker ‘_’ and all tonal diacritics are represented by numbers and follow the right diacritic marker. So, for example, a long unrounded open-mid front vowel associated with high tone would be symbolized as ‘io_l—1’.
So configured, this transcription system is designed to be usable from both a processing and a human-factors perspective. Because the symbol string is structurally defined and delimited, and because its symbol inventory lacks potentially ambiguous signs in a programming environment, these transcription strings are easy to parse and manipulate. Because individual symbols are consistently used within their type (consonant or vowel) and class (position number) and were chosen to be as congruent as possible with symbols from pre-existing phonetic symbol inventories, and because string syntax is well defined, this transcription system is relatively easily learned and expanded (unlike, for example, the IPA system).
As noted above, each symbol string preferably consists of a two character base sign to which diacritic markers may be suffixed. Suffixes only terminate in either the right-frame diacritic marker ‘_’ or a right frame diacritic marker followed by one or more numerals. In this embodiment, the first position character of a base sign always marks the string as either consonant or vowel, i.e., there is no overlap of characters between the first position consonant and vowel symbol inventories. Finally, there is no use of symbols that are employed in a special way in programming. Thus, every symbol-string label indexing a distinct phone can be easily identified and parsed with minimal programming effort, and it is not generally necessary (as it is with many other phonetic alphabets) to build a special-purpose scanner designed to convert these labels to other label types (numbers, for example).
Each individual character used in the above described preferred transcription system comprises a sign, or form-meaning pair. The relationship between a character's graphic form and its phonetic meaning is one-to-many and uniquely dependent on the sign's context, that is, its place in the symbol-string label. For example, the form ‘w’ is associated with three distinct articulation features or feature constellations: 1) voicing and approximation, 2) pre-aspiration, and 3) labial offgliding. Notwithstanding the above, the meaning of any particular ‘w’ is never ambiguous: ‘w’ is associated only with meaning (1) as a second position consonant sign; it indexes meaning (2) as a third position consonant diacritic; and as a third position vowel diacritic, it is uniquely linked to meaning (3).
Most alphabetic characters are strongly associated with phonetic meanings that users literate in Romanized orthographies learned during childhood. Because of this, there is an attempt in most multilingual phonetic inventories to conform as much as possible to pre-existing form-meaning associations, and the above described preferred transcription system mirrors this viewpoint. Thus the symbol ‘i’ is typically associated with the vowel features ‘close’, ‘front’ and ‘unrounded’ and this proposed transcription system observes this conventional sense as well. Only these features are divided between two classes: vowel position 1 (where ‘i’ means ‘front’ and ‘unrounded’) and vowel position 2 (where it means ‘close’). By unpacking a character form's typically associated feature set, it becomes possible to use the form in several distinct signs, each associated with just one or two features of the source sign.
There are presently several pre-existing phonetic alphabets in common use within the speech research community. XSAMPA and Worldbet, for example, are relatively widely employed for multilingual applications (though again, these systems tend to suffer drawbacks associated with computer processing applications and conflicts). Appendix A attached hereto presents an abbreviated base-sign inventory for the preferred transcription system (denoted as “new” in the appendix) described above and further maps those contents to the XSAMPA, Worldbet, and the IPA phonetic alphabets for the convenience of the reader.
With continued reference to
In a preferred approach, the polyphone network includes and/or comprises a hierarchical phone correspondence network. In one embodiment, that hierarchy can be configured as a plurality of merged phone selections organized in at least one tree (such as, for example, a binary branching tree hierarchical arrangement). For example, and referring now to
In a similar fashion, the resultant phone categories (and remaining unmerged phones of the first hierarchical level 21) are again merged as and how appropriate at a next yet-lower-resolution hierarchical level 23. For example, PHONE 11 and PHONE 4 are merged at this level to yield PHONE 12. In a preferred embodiment, fifteen such levels are provided (with the intervening levels being indicated by the ellipses 24 depicted), with the final lowest resolution view 25 preferably comprising a final category PHONE XY. Conceptually, this final hierarchical level 25 comprises a single sound that purports to grossly characterize the entire polyphone network,
So configured, these various views provide phone categories that correspond to views of varying resolution wherein the phone information essentially represents, at least in part, cross-linguistic and language-independent mutual phonetic similarities and non-contrastiveness of corresponding higher resolution phone information views. As the degree of resolution drops by moving higher in the tree, at least some of the merged phone nodes each comprise a merger of a plurality of phone categories, which phone categories are themselves each most representative of a given plurality of phone categories.
For many tasks, of course, it is not feasible to use a narrow phone transcription as represented at the highest resolution levels of the suggested polyphone network 20. All languages collapse distinct phones that are non-distinct phonologically into phonemes. Using the preferred transcription system detailed above, consider that in English, for example, a flapped alveolar stop (td), a voiceless alveolar stop (tp) and a voiceless aspirated alveolar stop (tp_c_) are all allophones of the phoneme represented by the English alphabetic symbol ‘t’. Likewise, it is possible for distinct phonemes across languages (though not distinct phonemes internal to a language) to be collapsed into a merged ‘superphoneme’ classification system. Thus, for example, it is usually the case that a voiceless dental stop phoneme (dp) and a voiceless alveolar stop phoneme (tp) may share acoustic models without losing crucial cross-linguistic phonological distinctions. Such knowledge can be exploited when reducing a multilingual model inventory to a size that is more compatible with recognition systems in embedded devices.
The polyphone network proposed herein comprises a phone merger specification tool for shared multilingual and multi-dialect acoustic modeling that takes advantage of phonetic similarities (as inherently and conveniently defined by the preferred transcription system described above) while maximally preserving known phonological contrasts. In a preferred embodiment, the polyphone network comprises a binary branching tree that has as its leaves a nearly exhaustive inventory of core phones derived from a rich set of typologically diverse languages (where preferably at least all major language families are represented) (and wherein core phone should be understood to refer to an allophone that is most representative of a particular phoneme, defined cross-linguistically). Lower resolution nodes on the tree represent superphonetic categories with merger being preferably based entirely on phonetic similarity, core phone frequency, and language-independent (universal) phonological laws and tendencies.
As an illustration of how language-independent factors can contribute to such a hierarchical arrangement of phone classes, a detailed example is provided below:
Consider the following set of language universals, tendencies, and frequency observations:
Language universal (1): all languages have stop consonants;
Language tendency (1): languages tend to have a plain voiceless stop consonant series;
Frequency observation (1): p, t and k are the most frequently occurring stop consonants in the world's inventories, with t being the most common and p the least common of the three;
Language universal (2): t<k<p; and
Language tendency (2): language phoneme inventories exploit the maxim of sufficient discriminability—that is, contrastive sounds in any given language must be sufficiently distinct from one another to be distinguishable in articulation and perception.
The first language universal points to the relative importance of stops as a consonant class. Other consonant classes (fricatives, liquids, nasals, and so forth) are not universally present in the world's sound inventories. Thus among the consonants, stops are preferably ranked higher than the other classes. This indicates that the stop class is the highest level consonant class in the phone hierarchy. Language tendency (1) provides additional information about stops. in addition to whatever other stop series a language may have—voiced, ejective, implosive, and so forth—it is likely to have a plain voiceless series. This means that within the stop class, the plain voiceless series is ranked highest in relative cross-linguistic importance.
The provided frequency observation records the overall importance of the three plain voiceless stops p, t, and k. These three stops occur more frequently than any other stops in the world's inventories. And within this set, there is a frequency hierarchy, where t is ranked the highest. The second language universal, t<k<p, signifies that if a language has the stop p, it must also have the stops k and t; and if a language has k, it necessarily has t.
Finally, the second language tendency suggests that contrastive sounds in a given language tend to be distributed proportionally among a range of articulators (lips, alveolar ridge, palate, velum, larynx, and so forth) and to not cluster together at the same region. Therefore, for example, no language contrasts palato-alveolar and palatal phonemes within a given manner series, and contrasts among dentals and alveolars within the same manner series are extremely rare. Notice that the three most common plain voiceless stops, p, t, and k observe this maxim; p is a bilabial stop, t is a dental or alveolar stop, and k is a velar stop.
These above factors contribute to the following preferred high-level hierarchical arrangements for stop consonants in the proposed polyphone network (set forth in the notation of the earlier described preferred transcription):
At each node in the tree a confidence score between ‘1’ and ‘10’ appears in square brackets, where ‘10’ indicates very high confidence in the merge and ‘1’ indicates very low confidence in the merge. In the above examples all nodes have a confidence score of ‘2’. These scores are preferably derived from three knowledge sources: 1) relative phonetic similarity (defined automatically by the transcription structure), 2) relative phonological contrastiveness (that is, how often do the phones included in the children nodes contrast phonologically), and 3) frequency (that is, how frequently the phones included in the children nodes occur cross-linguistically).
The above example illustrates how the preferred polyphone network provides a novel, linguistically informed framework within which the phone model inventory size and search space issues inherent to embedded multilingual or multi-dialect automatic speech recognition can be accommodated while minimizing the negative performance impact of the resource restrictions. The complete proposed preferred polyphone network appears in Appendix B attached hereto.
Such a polyphone network is well suited for multi-language phone transcription, and especially for computational applications. Such a network explicitly encodes phonetic features that both make the inventory maximally extensible and contributes to automatic phonetic similarity metrics in the polyphone network. It can be seen that such a polyphone network further comprises a universal phone merger tool that surpasses other strategies for phone merger by taking into account not only phonetic similarity, but also language-independent universals. Using such a polyphone network, it is possible to derive a language-inclusive phone inventory which has a minimal number of distinct units but which maximally preserves both language-internal and cross-language contrasts. Use of such an inventory greatly reduces corresponding speech recognition search efforts.
As illustrated in
Those skilled in the art will recognize that a wide variety of modifications, alterations, and combinations can be made with respect to the above described embodiments without departing from the spirit and scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6085160 | D'hoore et al. | Jul 2000 | A |
| 6912499 | Sabourin et al. | Jun 2005 | B1 |
| Number | Date | Country | |
|---|---|---|---|
| 20040176078 A1 | Sep 2004 | US |
