Patent Grant
11776533
The technology disclosed relates to natural language understanding (NLU) and to analysis of meaning of text and spoken phrases. In particular, it relates to new programming constructs and tools and processing patterns that implement those new programming constructs.
Conventional programming for natural language understanding is arcane and requires great expertise, as shown by
Other grammar-based NLU frameworks, such as those by Nuance, are built around defining a fixed set of slots that represent the expected information supplied in utterances within a vertical application and determining how each phrase in the grammar results in filling those slots. See, e.g., Grammar Developer's Guide, Nuance Speech Recognition System Version 8.5, Chapter 4 (2003). Use of grammar slots is consistent with the W3C standard known as Voice XML. See, e.g., Scott McGlashan et al., Voice Extensible Markup Language (VoiceXML) 3.0, section 6.10 Field Module, Table 41 (8th Working Draft December 2010). Version 2.0 of the VoiceXML recommendation has been implemented in BeVocal's Nuance Café, with grammar slots. See, BeVocal, VoiceXML Tutorial, p. 49 Grammar Slots (2005); BeVocal, VoiceXML Programmer's Guide (2005); BeVocal, Grammar Reference (2005). Like GF, VoiceXML involves multiple layers of abstraction and special expertise. In this context, a vertical application or vertical market application, is software defined by requirements for a single or narrowly defined market. It contrasts with horizontal application. An example provided by Wikipedia of a vertical application is software that helps doctors manage patient records, insurance billing, etc. Software like this can be purchased off-the-shelf and used as-is, or the doctor can hire a consultant to modify the software to accommodate the needs of the doctor. The software is specifically designed to be used by any doctor's office, but would not be useful to other businesses.
Custom applications of NLU have gained momentum with the introduction of Apple's Siri voice recognition on the iPhone 4S. However, at this time, the API to Siri is not publically released.
An opportunity arises to introduce authoring technology for custom applications of natural language understanding. More efficient, reliable, updatable and maintainable custom applications may result. More custom applications may be produced, as barriers to entry for authors are lowered. Sharing of NLU blocks that understand common semantic units that are frequently used, such as dates, times, location, phone numbers, email address, URLs, search queries, etc. can be expected as the pool of NLU authors expands greatly. Improved user interfaces may ultimately be produced.
The technology disclosed relates to authoring of vertical applications of natural language understanding (NLU), which analyze text or utterances and construct their meaning. In particular, it relates to new programming constructs and tools and data structures implementing those new applications. Other aspects and advantages of the technology disclosed can be seen on review of the drawings, the detailed description and the claims, which follow.
A detailed description of implementations of the technology disclosed is provided with reference to the
In state of the art implementations of systems that utilize speech recognition and natural language processing, speech recognition is typically applied first to produce a sequence of words or a set of hypotheses. Sometimes, this speech recognition is referred to as a combination of acoustic recognition and language, or linguistic, recognition. Speech recognition output is sent to the NLU system to extract the meaning. The weakness of such a system is that errors in the speech recognition output can cause the NLU system to misunderstand the intended meaning. To quantify how severe this problem may be, consider the example of a speech recognition system that has ninety percent word accuracy. This system could have zero percent sentence accuracy for a large number of sentences, such as sentences that are 10 words long, with one word being wrong in each sentence. Incorrect sentences can have the wrong meaning or no meaning at all from the point of view of a NLU system.
The technology disclosed includes an integrated approach that decodes both speech and meaning concurrently. If at any time interval, a partial hypothesis is not a prefix of a parsable phrase, it can be eliminated or pruned from a set of token recognition hypotheses for the speech recognition. For those hypotheses that are parsable prefixes, the partial score of the parse can be accumulated, influencing the more likely phrases from a meaning parser's perspective to have a higher score for speech recognition. This approach increases accuracy by eliminating meaningless phrases and by promoting more likely phrases. This approach also improves latency. Response time decreases due to the fact that both steps are executed concurrently as opposed to sequentially.
The technology disclosed applies the run time meaning parser to new programming constructs that extend a programming language, such as a general-purpose programming language, using the interpret blocks and interpret statements. The interpret blocks process token lists (or phrases), produce scores for pattern matches and incomplete pattern matches, and return meanings associated with completed pattern matches. The interpret statements identify meaningful patterns that can be constructed from one or more words in a natural language, extended pattern tables, custom statistical language models and other interpret blocks. Interpret blocks also link interpret statements with action statements written in the general-purpose programming language which can be invoked by a completed pattern match to perform actions programmed using the power of the general-purpose programming language.
Introductory Example Using Extended Pattern Tables
Before turning to the figures, it is useful to consider an example that introduces the subtlety and depth of language understanding problems and the power of the technology disclosed to solve many such problems. The first example, which is further discussed below in the context of
Imagine the user inputting the query, “one twenty first avenue san jose”. At first glance, this query could map to a number of possible addresses such as:
The list of possible options is actually larger. For example, even though “san jose” is a famous city in California, there is also a city with the same name in New Mexico and another one in Illinois. The developer could potentially decide to distinguish among the options by considering the location of the user, as well as whether the number in the address specified is a valid one, by executing programming statements at various parsing states.
Furthermore, it is likely that the correct target is actually “first street”, but the user makes a mistake and calls it “first avenue”. Therefore, the developer may want to allow using alternative common suffixes, or skipping the suffix completely.
Finally, if the street has a direction, such as “N First Street”, the user may or may not include the word “north” in the query, and if included, may put it in the wrong place as in “first street north” instead of “north first street”.
Given the database of street addresses and their valid street number ranges, using extended pattern tables, interpret blocks and action statements, all of the above can be achieved efficiently in this system.
One implementation would start with the following interpret block that includes an interpret statement with an extended pattern table that, when the matches are returned, uses an auxiliary table to look up valid street addresses for particular streets:
public block (string full_address) US_STREET_ADDRESS( ){
n_street=US_STREET_ADDRESS_TABLE( )} as {
}
the weights in this interpret statement are explained below.
The optional STREET_NUMBER( ) block captures a number in the address and returns its value as a street address number. This block can process input tokens that are as simple as a digit sequence, which, once defined can be easily used, or it can be extended to support more complex cases such as street numbers with dashes, fractions or letters of the alphabet. The modularity of this block supports incremental extension from simple to more complex cases.
The next entity is a more detailed example of an extended pattern table US_STREET_ADDRESS_TABLE( ) that can represent all the variations of all the street addresses that the developer intends to support. This table is sometimes called an extended pattern table in contrast to bigram and trigram sequences in conventional statistical language models. An extended pattern table is used to parse the meaning of a token sequence, rather than predict the likelihood that a speaker or typist would utter or enter the token sequence. Here are a few sample rows of such a street name extended pattern table, which also appear as rows in the table in
table (unsigned id, string name) US_STREET_ADDRESS_TABLE [
]
This table has several notable characteristics. First, the table returns two values: an “id”, a unique number which points to a data structure that contains or refers to information about each street, such as a list of valid street number and the full name of the street as a string. The street name alone is provided here as an example of multiple return values. Other information such as the street's city, state and geo coordinates can also be part of the data structure that the id points to.
Second, “N first street” has the direction “north” before the street name. In the table above the developer allows the presence of “north” to be skipped with a penalty. Since there is a weight of 10 before the optional “north”, written as [10 “north”], the interpret pattern parser normalizes skipping north to have a probability of 1/(10+1)=1/11, while presence of north has the probability of 10/11. Alternatively, “north” can come after the street name, with a weight of 1/5, which again is normalized by the interpret pattern to have a probability of 1/(1+1/5), since sum of weights need to add up to 1. For each variation where “north” appears, the “street” suffix can be skipped completely, but with a weight of 1/51. If a suffix is provided, it can be the correct suffix “street” with the weight of 1/(1.1). Common alternatives of “avenue” and “road” have a joint weight of 0.1/1.1=1/11. Since there are 2 alternatives, each gets a weight of 1/2 of that, i.e., 1/22. A modified regular expression style syntax can used as seen with weights prefixing speech recognition results (or text inputs) as quoted strings combined with symbols such as pipe (“|”), parenthesis (“( )”), brackets (“[ ]”), and period (“.”) used to express alternatives, grouping, optionality, and conjunction. In one embodiment, weightings are automatically normalized and so the table expression to recognize any one of a, b, or c: (10 “a”|5 “b”|“c”), would have normalized weights of 10/16 “a”, 5/16 “b” and 1/16 “c”. Note that a weight of 1 was assumed for “c”.
Third, this table includes “N First Street”, “1st street” and “21st avenue”, which all match the query example above. Assuming no other row of the table matches that query example, then the table entry that is pointed to by “N First Street”, returns 2 rows of the table for when “one twenty” is matched to the street number, and 1 row of the table when “one” is matched to the street number. The action statements in the programming language inside the interpret block can then evaluate various conditions, including the weight of the matched row, the range of street numbers in that row, the current location of the user and the distance from that location to the location of the matched row, to pick at least one result. See below discussing
Here is an example of the action statements that perform the logic inside the interpret block, written in C++. From the US_STREET_ADDRESS( ) interpret block we obtain the variable n_number which is set to a street number, if one is present, and the variable n_street, which points to an entry in an extended pattern table of US streets. In the specific implementation, n_street is the head of a linked list of specific streets, each with a full street name and other properties. The code below determines the street with the best weight among the candidate streets. When performing this comparison, it takes the street number into account by calling an auxiliary function to check if a particular number is valid on a particular street. Here is are the action statements for the US_STREET_ADDRESS( ) block:
The program above can be extended to include additional logic such as the user location (see
Figures Discussed
In one implementation, the network 135 includes the Internet. The network 135 also can utilize dedicated or private communication links that are not necessarily part of the Internet. In one implementation the network 135 uses standard communication technologies, protocols, and/or inter-process communication technologies.
Applications are used with the phrase interpreter engine 117 to understand spoken or text input. The client computing devices 155, the application parser engine 130 and the phrase interpreter engine 117 each include memory for storage of data and software applications, a processor for accessing data in executing applications, and components that facilitate communication over the network 135. The computing devices 155 execute applications, such as web browsers (e.g., a web browser application 157 executing on the computing device 156), to allow developers to prepare and submit application code 140 and allow users to submit phrases to be interpreted by the phrase interpreter engine 117. The computing devices 155, 156 may be for example a workstation, desktop computer, laptop, a tablet computer, mobile phone, or any other type of computing device.
The application parser engine 130 receives applications and parses them, producing a parse tree or an event stream. It produces application data structures 120 from the parsed application code 140. An application data structure 120 may represent a single application 140. Alternatively, multiple applications 140 may be used to prepare a single application data structure 120. The application data structure can, for instance, be a tree, a state machine, or a network of valid tokens. The application data structure can be compiled or in an interpretable structure. The application data structure 120 can include nodes that reference the custom SLMs 110 and the extended pattern tables 115. This data may be stored collectively or in copies on multiple computers and/or storage devices.
The acoustic-language recognizer 128 can be a conventional acoustic or speech recognition component that outputs tokens. It can operate in one or more stages. In this application, an acoustic-language recognizer or processor 128 can potentially only include a single stage, acoustic recognition-only processing without application of separate linguistic analysis. The technology disclosed can be applied to coupling preliminary processing to meaning-specific patterns, even when the tokens from preliminary processing are phonemes or other tokens that are not full words. For instance, an acoustic stage can process input sound samples to produce phonemes. These phonemes can be passed to a language or linguistic stage that considers and scores sequences of phonemes. Language recognizers sometimes use diphone or triphone analysis to recognize likely sequences of phonemes. Language recognizers sometimes use statistical language models to recognize statistically likely sequences of words.
Phrase interpreter engine 117 includes an acoustic-language recognizer 128 and the meaning parser 118. The phrase interpreter engine 117, like the application parser engine 130, is implemented using at least one hardware component. The engines are implemented in hardware, firmware, or software running on hardware. Software that is combined with hardware to carry out the actions of a phrase interpreter engine 117 can be stored on computer readable media such a rotating or non-rotating memory. The non-rotating memory can be volatile or non-volatile. In this application, computer readable media does not include a transitory electromagnetic signal that is not stored in a memory; computer readable media stores program instructions for execution.
A smart program editor 210 can recognize the programming constructs and keywords corresponding to the technology disclosed. It can color code or otherwise highlight the keywords. It may check the syntax of the programming constructs disclosed as a programmer types. It can create stub code for structures to be completed by the programmer. While smart editors have been used for many years, the programming constructs disclosed herein are new to natural language understanding programming, and offer new opportunities for developer support.
Regular editors also can be used to author electronic records including program code. When a regular editor is used, a pretty printer that recognizes the disclosed programming constructs can be used to format the code to make it more readable. Code formatting is often supported in CASE tools, IDEs and smart editors. Still, there are standalone code pretty printing products, not shown in
A parser 221 receives one of more records 211 and converts them to machine recognized format. One machine recognized format is a parse tree. Another machine recognized format is a stream of events.
An interpreter or compiler 231 uses the output of the parser. The interpreter typically uses the parser output directly in combination with a runtime 241 to execute or debug a program. An interpreter may persist an intermediate format for execution, debugging or optimized compilation. The intermediate format may be programming language independent.
A compiler typically uses the output of the parser to compile object code or pseudocode. Object code can be optimized for particular platform. Pseudocode can be machine independent and can be run on a virtual machine, instead of directly on a physical machine.
A preprocessor also may use output from the parser to expand the disclosed programming constructs into code in the programming language before its interpretation or compilation. A preprocessor may be integrated with a parser.
A variety of devices 235 are indicated which may be targets for NLU development. These devices may include handheld devices, such as smart phones and tablets, mobile devices such as laptops and workstations or PC. In addition, NLU components can be deployed to servers 237 coupled in communication with other devices 235.
In the alternatives and combinations described, there are many environments and many implementations in which the technology disclosed can be practiced.
In
In
When processing spoken input, the acoustic language recognizer 128 produces token phrases and alternative token recognition hypotheses 437. At each time interval during interpretation of spoken input, hundreds or even thousands of token recognition hypotheses 437 can be generated. To expand on this point, an interval such as every 10 or 100 milliseconds could be selected to test and generate token recognition hypotheses. At each interval, a thousand, three thousand, or an even larger number of hypotheses could be generated. In some implementations, enumeration of hypotheses could exhaust all combinatorial possibilities for hypotheses.
The conventional acoustic-language recognizer scores the alternative token recognition hypotheses that it produces and selects a set of token recognition hypotheses for further processing. A two-stage acoustic-language recognizer arrangement, an acoustic recognizer applies an acoustic recognition score to alternative token sequences, selects the set of token sequences and sends them to a language or linguistic recognizer. The language recognizer applies a language model, such as a statistical language model, and scores the token sequence hypotheses. It returns the scores to the acoustic recognition stage, which combines the acoustic and language recognition scores.
In general, the acoustic-language recognizer 128 sends token recognition hypotheses 437 to the meaning parser 118 as token phrases, sequences, or lists 438. The meaning parser 118 processes the tokens and returns values 438. Complete and incomplete parses of tokens can be scored by the meaning parser 118 to return meaning recognition scores. Unrecognizable token phrases can be flagged as such in the returned values. Completed parses of token phrases that satisfy a recognized pattern can further return data and/or references to data structures that express the meaning of the token phrase. Within the processing structure of the meaning parser 118, a token phrase being processed can be all or part of the token recognition hypothesis 437.
One implementation of meaning parser 118 includes a token processor 455, table handler 465, an SLM handler 475, and a scorer 485. Some implementations may have different and/or additional modules than those shown in
A scorer 485 accumulates and normalizes a meaning recognition score during processing of a token phrase. The scorer 485 can generate scores for both partial and completed pattern recognition. Scoring of token sequences against patterns is the subject of
Meaning parser 118 further executes action statements contained within interpret statements. These action statements are discussed above in the context of the introductory example and below in the context of
Weights are indicated in two ways. Inside curly braces, alternatives are separated by the symbol |, which means “or”. Integers or floats can be used to indicate the relative weights of the alternatives. For instance, the pattern in lines 723, 725 implicitly assigns a weight of one to the part of the pattern at nesting level 2 in 723 and a weight of 1/5 to nesting level 2 in 725. The weight of one is implicitly assigned because no explicit weight is given for nesting level 2 in 723. The weight given in curly brackets at the beginning of line 723 expresses the ratio between alternatives as “{5|1}” to indicate that the first alternative in line 723 has five times as much likelihood and weight as the second alternative in line 725. A second type of weight is indicated by fractions without curly brackets, tracking optional pattern elements that appear in square brackets. For instance, in line 723, the pattern element “north” is indicated as optional within square brackets. The weight 10 precedes the token. This indicates that it is 10 times as likely that the token “north” will precede “first” in a reference that means “N. 1st Avenue”, as it is likely to be omitted. When the token “north” appears in the token phrase, this term in the pattern is given a weight of 10/11. When the token is omitted, this term in the pattern is given a weight of 1/11. With this explanation in mind, the following scoring examples can readily be understood.
In
The token string does not include the optional word “in”, which is weighted as unlikely to be used. The omission of this token effectively has a weight of 10/11. The token string also omits the state, which is equally likely to appear be omitted, so a weight of 1/2 is applied to the omission.
The partial scores illustrated in this figure can be combined by multiplying them together. Each of the partial scores is between zero and one. The product of the scores also will be between zero and one. In another implementation, an average of the scores could be calculated, which also would be between zero and one. Both multiplicative and additive approaches can be used, for instance, multiplying together weights in a pattern segment (between two periods) and taking the average of weights across the pattern segments.
In
In
These tables, with extended patterns or simple patterns, can be combined with statistical language models as illustrated by the following example.
It is quite possible for a sequence of input tokens to have more than one parse (or partial parse), with each parse having its own weight. The weights for all of the partial parses can be combined to represent the total weight (or probability) of a sequence of tokens. The weight of a sequence of tokens is useful and can be used, for example, as the score of the sequence of tokens in a speech recognition engine where multiple sequences of tokens are being considered. In one implementation, the total weight is the sum of all the weights, which makes statistical sense as weights are considered to be probabilities (the probability of a sequence of tokens is the sum of the probabilities of all its possible parses). In another implementation, the maximum weight can be used. In yet another implementation, the average of the weights can be calculated. To give an example of an implementation that adds the sum of multiple parses (or multiple options returned by a table) consider the table of
Further Email Example Using SLM Pattern
In another example, an interpret block is used with a table and two custom statistical language models to compose an email message. The table is a simple table of contacts. The SLMs are for subject and message.
Consider the following pattern portion of an interpret statement:
interpret (“email”. CONTACT( ). “subject”. SUBJECT( ). “message”. MESSAGE( )}
In the above example, CONTACT( ) is a table pattern representing a list of valid contacts in a database that the user can send emails to. SUBJECT( ) represents a statistical language model, while MESSAGE also represents a statistical language model, and the developer can choose to use a different language model for each. This makes sense as the statistical properties of an email subject could be different from that of the body of an email.
Given the query, “email ernie subject lunch meeting message i am running late”, the system matches “ernie” as the recipient (assuming Ernie is in the CONTACT( ) database), the subject becomes “lunch meeting” and the body of the email becomes “i am running late”. The system scores the subject line using the statistical language model represented by SUBJECT( ), and the body of the email using the statistical language model represented by MESSAGE( ). The system can do that because as it is parsing the query, it knows the state of its various parses. Now assume the following query:
“email ernie subject lunch meeting message message me when you get here”
The above query can be interpreted in multiple ways, most notably whether the word “message” is part of the subject:
Although both of these parses are valid, if the statistical language models are trained properly, the second parse should hopefully have a lower score, since “me when you get here” is a less likely body of a message.
The ability to switch between blocks, tables, and multiple statistical language models depending on the state of the parse, and execute programming statements at any point during the parse, makes the system a very powerful tool that can be used to implement a wide variety of tasks.
Discussion of Code with Interpret Blocks and Interpret Statements
The code in
The technology disclosed can be used to extend a programming language by the addition of a small number of constructs. Alternatively, the technology can serve as a core around which a programming language is designed. In some implementations, the technology disclosed is used to extend a mainstream, well-understood programming language such as C++. In other implementations, it may extend other programming languages such as Objective C, C#, Javascript, Java, Python, Ruby, or Perl. Extending a general purpose programming language takes advantage of an existing large pool of programmers who already are familiar with the language. The technology disclosed also could be incorporated into a special purpose programming language with benefits to those who understand or choose to learn the special purpose language.
The technology disclosed juxtaposes patterns and actions in two programming constructs, which we call interpret-blocks and interpret-statements. Interpret-statements such as 913, 914, 928 include both a pattern and an action to be performed (or action statements) when the pattern matches an input text or utterance. The patterns in this example are modified regular expressions of terminal and non-terminal symbols, with optional weights. For instance, in interpret-statement 928 of
In one page and a little more of
To summarize,
The exclude-statement 1013 stops execution of the block without processing subsequent interpret-statements and without assigning values to the block variables 1012, when the input includes phrases that indicate recurring dates, as this block does not handle recurring dates. In one embodiment, exclude statements are a type of interpret statement.
Interpret-statements 1014, 1015, 1016 handle the words “today”, “this”, “tomorrow” and “day after tomorrow”. Optional words that do not change the selected date include approximate times of day, such as “morning”, “evening”, “afternoon” etc. Some of these optional words may trigger assignment of a positive value to the variable “pm_hint”. Parts of the pattern in interpret-statement 1014, for instance, cause values to be assigned to n1 or n2. If values have been assigned to n1 or n2, the last line in 1014 resets pm_hint from 0 to 1. In various other interpret-statements, several qualitative block variables 1012, such as month_index_implicit, year_index_implicit, week_delay and pm_hint are given values.
Interpret-statements 1021, 1026, 1031, 1033 and 1036 include weights 1022, 1027, 1032, 1034, 1037 assigned when optional approximate times of day are part of the input. Four of the five interpret-statements use the same weight. The fifth 1033, assigns a much smaller weight 1034. These weights can be used by a runtime system to select among multiple interpret statements that may be triggered by a particular input pattern. In some implementations, the parser automatically normalizes weights returned by interpret-statements and parts of interpret-statements. Normalizing weights to total one another chosen value simplifies human understanding of output and debugging output when the example code runs. Normalizing weights to sum to one effectively produces a Bayesian probability distribution that can be used at runtime.
The DATE( ) example in
The main programming constructs in
In
Another example in
Overall, the code in
Multiple interpret-statements can be invoked from a single interpret block to combine a variety of applications with a common entry point. An example follows:
public block(CommandSpec command) ALL_COMMAND( )
{
};
This is one way to implement an aggregated natural language library or sub-library, as discussed in paragraphs 0085-0126 of US 2012/0233207A1, which is incorporated by reference above. More specifically, each of the above commands could have been independently developed by different individuals and entities and tied together by the common entry point by yet another developer. This creates a platform for an ever-improving system to which many developers can contribute and which many can help maintain. With this platform approach, each domain or vertical can be created or maintained by experts in that domain. For example, weather service providers can wok on the weather vertical, while navigation is created and maintained by other experts and so on.
Some Particular Implementations
In one implementation, a method is described that includes an automated method of accurately recognizing a speaker's meaning. This method includes producing and scoring partial transcriptions of a spoken phrase at intervals as the phrase is spoken using at least one acoustic-language recognition stage and a meaning parser stage. Practicing this method, at second and subsequent intervals, the acoustic-language stage generates a multiplicity of token recognition hypotheses that build on prior partial transcriptions and selects a set of the token recognition hypotheses using at least prior scoring of the prior partial transcriptions at earlier intervals and current acoustic-language scoring of the token recognition hypotheses at a present interval. The meaning parser stage concurrently processes particular token recognition hypotheses in the set of the token recognition hypotheses; determines whether a particular token recognition hypothesis has a parsable meaning; rejects unparsable hypotheses; and scores and returns at least one parsable meaning score for a particular token recognition hypothesis that has a parsable meaning. The acoustic-language recognition stage further stores for use at subsequent intervals combined scores of the token recognition hypotheses for current partial transcriptions, the combined scores incorporating at least the acoustic-language recognition scores and the parsable meaning scores.
This method or other implementations of the technology disclosed can each optionally include one or more of the following features. The acoustic-language recognition stage can prune the prior partial transcriptions used to build the token recognition hypotheses using the meaning parser stage rejections of the unparsable hypotheses. At an apparent end of the spoken phrase, the method includes selecting at least one completed transcription of the spoken phrase that has been scored as recognizable by the acoustic-language recognition stage and as parsable by the meaning parser stage.
The meaning parser stage can further take the actions of processing the token recognition hypothesis that includes at least one ambiguously expressed element and at least one dependent element that correlates with the ambiguously expressed element against an interpretation pattern and of processing and scoring the ambiguously expressed element against multiple rows of an extended pattern table that is invoked while processing the interpretation pattern, at least some of the rows declaratively expressing weighted alternative expressions of a particular ambiguously expressed element.
Handling ambiguously expressed elements further include applying logic expressed in a general programming language to process and rescore at least some of the scored rows from the extended pattern table using return values from the scored rows in combination with against at least information from a group consisting of (1) the dependent element from the token recognition hypothesis, (2) an optional element in the token recognition hypothesis, and (3) supplemental information not included in the token recognition hypothesis.
One implementation can further include applying logic expressed in a general programming language to process and rescore at least some of the scored rows from the extended pattern table, including comparing valid dependent values in an auxiliary table with the dependent element from the token recognition hypothesis.
One implementation can further include applying logic expressed in a general programming language to process supplemental information not included in the token recognition hypothesis and rescore at least some of the scored rows.
One implementation can further include applying logic expressed in a general programming language to process and rescore at least some of the scored rows from the extended pattern table against optional elements in the token recognition hypothesis.
One implementation can further include the meaning parser stage scoring the token recognition hypotheses against an interpretation pattern that includes at least one predicate condition and at least one statistical language model applied when the predicate condition is satisfied.
In some implementations, wherein a particular token recognition hypothesis includes an ambiguously expressed element and a dependent element that correlates with the ambiguously expressed element; the meaning parser stage further takes the actions of scoring the token recognition hypothesis against a plurality of interpretation patterns built from a model pattern, the model pattern implementing at least: (1) a table pattern that includes rows in an extended pattern table, at least some of the rows declaratively expressing weighted alternative expressions of a particular ambiguously expressed element; and (2) a statistical pattern that includes a predicate condition and a custom statistical language model applied when the predicate condition is satisfied. In these implementations, applying the table pattern includes scoring the token recognition hypothesis against multiple rows of the extended pattern table. In addition, applying the statistical pattern includes scoring the token recognition hypothesis against the custom statistical language model.
Processing results of scoring the tech in recognition hypothesis against rows of the table can further include applying logic expressed in a general programming language to process and rescore at least some scored rows from the extended pattern table using at least information from a group consisting of: (1) the dependent element from the token recognition hypothesis; (2) an optional element in the token recognition hypothesis; and (3) supplemental information not included in the token recognition hypothesis. This processing also can include applying the logic expressed in the general programming language to process and rescore at least some of the scored rows from the extended pattern table against valid dependent values in an auxiliary table and the dependent element from the token recognition hypothesis.
It can include applying the logic expressed in the general programming language to process and rescore at least some of the scored rows from the dependent value table against the optional element in the token recognition hypothesis.
It can include applying the logic expressed in the general programming language to process the supplemental information not included in the token recognition hypothesis and rescore at least some of the scored rows.
Other implementations may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform a method as described above. Yet another implementation include a system including memory and one or more processors operable to execute instructions, stored in memory, to perform a method as described above.
In another implementation, a method is described that, in some environments accurately recognizes an intended meaning of a complete phrase. This method includes invoking an interpretation pattern that expresses a complete phrase with a meaning as a sequence of elements that include one or more words from a natural language and a table element that invokes an extended pattern table and receiving a text string of tokens that express an intended meaning, wherein the token include a combination of at least one ambiguously expressed element and a dependent element that correlates with the ambiguously expressed element, and further including one or more supplemental elements. Further includes generating a plurality of alternative interpretations of the combination of the ambiguously expressed element and the dependent element; and processing and scoring at least some tokens in the alternative interpretations against multiple rows of the extended pattern table, at least some of the rows declaratively expressing weighted alternative expressions of a particular ambiguously expressed element.
This method other implementations of the technology disclosed can each optionally include one or more of the following features.
The method can further include applying logic expressed in a general programming language to process and rescore at least some of the scored rows from the extended pattern table against at least information from a group consisting of (1) a dependent element from the text string, (2) an optional element in the text string, and (3) supplemental information not included in the text string.
In some implementations, the method further includes applying logic expressed in a general programming language to process and rescore at least some of the scored rows from the extended pattern table against valid dependent values in a dependent value table and a dependent element from the text string.
The method can further include applying logic expressed in a general programming language to process supplemental information not included in the text string and rescore at least some of the scored rows.
That it can further include applying logic expressed in a general programming language to process and rescore at least some of the scored rows from the dependent value table against optional elements in the text string; and selecting at least one intended meaning using at least the rescored rows of the extended pattern table.
Again, other implementations of this an additional methods described below also may include non-transitory computer readable storage medium storing instructions executable by a processor to perform a method is described. And another implementation may include the system including memory and one or more processors operable to execute instructions, stored in the memory, to perform a method is described. For the sake of brevity, the proviso in this paragraph is hereby applied the to the implementations in this section.
In another implementation of the technology disclosed, an automated method of building a natural language understanding application is described. The method includes receiving at least one electronic record containing programming code that interprets sequence of input tokens by extending a general purpose programming language with interpret-block and interpret-statement data structures. The interpret-block data structures include at least one of the interpret-statements and zero or more variables returned by the interpret-block. The interpret-statements include a pattern of one or more tokens, and zero or more action instructions. The action instructions perform logic not achieved by pattern matching and/or assign values to the variables of the interpret-block. The method further includes parsing the received program code to produce an executable representation of the interpret-block and interpret-statement data structures.
This method mother implementations of the technology disclosed can each optionally include one or more the following features.
At least one token of the interpret expression can be another interpret-block.
The returned parameters from other interpret-blocks are made available to the action statements inside the interpret block.
At least one token of the interpret expression can be a statistical language model. It also can be a wildcard. It can be a table of token expressions with fixed returned values for each row of the table and without any action statements. At least one sub-expression of the interpret expression is allowed to have repetitions. At least one of a minimum and a maximum number of repetitions of a sub-expression can be specified. The outgoing weights at each token can be normalized to add up to 1. The normalization of outgoing weights can be performed at sub nodes instead of tokens to reflect the way the expression is modularized.
Again, this method also can be practiced in a non-transitory computer readable storage medium or by a system.
In another implementation, a method is described that includes scoring a partial transcription of input. Practicing this method includes instantiating in memory at least one data structure derived from programming code that interprets token list using a general purpose programming language extended with interpret-block and interpret-statement data structures. The interpret-block data structures include at least one of the interpret-statements and one or more values returned by the interpret-block. The interpret-statements include patterns that are built from words in a target natural language, from at least one extended pattern table, and from references to additional interpret-blocks and action instructions in the general purpose programming language that are triggered by a match between parts of an input text and the patterns. The extended pattern table matches and scores at least part of the token list against multiple rows of in the extended pattern table, at least some of the rows declaratively expressing weighted alternative expressions of ambiguously expressed elements. The action instructions assign values to the variables of the interpret-block, which values attribute meaning to the token list. The method further includes receiving the token list and processing and scoring the token list against the data structure including scoring at least part of the token list against multiple rows of the extended pattern table, at least some of the rows declaratively expressing weighted alternative expressions of a particular ambiguously expressed element.
This method mother implementations the technology disclosed can each optionally include one or more of the following features. The action instructions can further include logic expressed in the general programming language to process and rescore at least some of the scored rows from the extended pattern table using at least information from a group consisting of: (1) a dependent element in the token list that has meaning in a context set by an ambiguously expressed element in the token list; (2) an optional element in the token list; and (3) supplemental information receive in addition to the token list.
The action instructions further can include logic expressed in a general programming language to process and rescore at least some of the scored rows from the extended pattern table comparing valid dependent values in an auxiliary value table with the dependent element.
The action instructions further can include logic expressed in a general programming language to process and rescore at least some of the scored rows from the extended pattern table using the optional element.
The action instructions further include logic expressed in a general programming language to process the supplemental information not included in the token list and rescore at least some of the scored rows.
Again, this method also can be practiced in a non-transitory computer readable storage medium or by a system.
In another implementation, method is described that includes building a natural language understanding (abbreviated NLU) data structure. This method includes receiving at least one electronic record containing programming code that interprets an input text by extending a general purpose programming language with interpret-block and interpret-statement data structures. The interpret-block data structures include at least one of the interpret-statements and one or more variables returned by the interpret-block. The interpret-statements include patterns that are built from words in a target natural language, from at least one extended pattern table and from references to additional interpret-blocks and action instructions in the general purpose programming language that are triggered by a match between parts of an input text and the patterns. The extended pattern table matches and scores at least part of the input text against multiple rows of in the extended pattern table, at least some of the rows declaratively expressing weighted alternative expressions of ambiguously expressed elements. The action instructions assign values to the variables of the interpret-block, which values attribute meaning to the text. The method further includes parsing the received program code to produce a data structure representing the interpret-block and interpret-statement data structures.
This method and other implementations of the technology disclosed can each optionally include one or more of the following features.
The pattern specified in the interpret-statement data structure can include a regular expression of the words and the additional interpret-blocks. The extended pattern table can be invoked by an antecedent event selected from a group at least consisting of: a match between part of the word hypothesis and at least one word in the natural language that is part of the pattern preceding the extended pattern table; and positioning of the extended pattern table as a first element of the pattern.
The general purpose programming language can belong to a “C” programming language family.
The set of the interpret-blocks collectively can define a vertical application of NLU.
Values assigned to the variables of a particular interpret-block can be available to additional interpret-blocks and to a NLU processor at run time.
Patterns in the interpret-statements in the set of interpret-blocks collectively can match substantially all of a vertical application vocabulary that is recognized by the vertical application of NLU.
The method can further include receiving a plurality of sets of the interpret-blocks that define a plurality of vertical applications and parsing the plurality of sets of interpret-blocks.
The interpret-block can further include at least one exclude-statement that contains an exclude pattern that is built from words in a target natural language and matching of the pattern in the exclude-statement causes an exit from the interpret-block without further processing of include-statements.
The patterns of the include-statements include relative weights assignable to matches of patterns or partial patterns.
Again, this method also can be practiced as code stored on a non-transitory computer readable storage medium or on running a system.
In another implementation, parser running on a processor is described that builds a representation of natural language understanding (abbreviated NLU). This parser includes, program instructions running on at least one processor that cause the processor to receive at least one electronic record containing programming code that interprets text or utterances by extending a general purpose programming language with interpret-block and interpret-statement data structures. The interpret-block data structures include at least one of the interpret-statements and one or move variables returned by the interpret-block. The interpret-statements include a pattern that is built from words in a target natural language or from references to additional interpret-blocks and action instructions in the general purpose programming language that are triggered by a match between parts of the text or utterances and the pattern. The action instructions assign values to the variables of the interpret-block, which values attribute meaning to the text or utterances. The parser parses the received program code to produce a parse tree that represents the interpret-block and interpret-statement data structures.
This parser in other implementations of the technology disclosed can optionally include one or more of the following features.
The pattern specified in the interpret-statement data structure can include a regular expression of the words and the additional interpret-blocks. The general purpose programming language belongs to a “C” programming language family. The values assigned to the variables of a particular interpret-block are available to additional interpret-blocks and to a NLU processor runtime. A set of the interpret-blocks collectively can define a vertical application of NLU.
The patterns in the interpret-statements in the set of interpret-blocks can collectively match substantially all of a vertical application vocabulary that is recognized by the vertical application of NLU.
Operation of the parser can further include receiving a plurality of sets of the interpret-blocks that define a plurality of vertical applications and parsing the plurality of sets of interpret-blocks. The interpret-block can further include at least one exclude-statement that contains an exclude pattern that is built from words in a target natural language and matching of the pattern in the exclude-statement causes an exit from the interpret-block without further processing of include-statements.
The patterns of the include-statements can further include relative weights assignable to matches of patterns or partial patterns.
Any of the methods described herein can be implemented as a computer-readable storage medium loaded with instructions that, when run on at least one processor, cause the processor to carry out the methods described. While the technology disclosed is disclosed by reference to the preferred embodiments and examples detailed above, it is to be understood that these examples are intended in an illustrative rather than in a limiting sense. It is contemplated that modifications and combinations will readily occur to those skilled in the art, which modifications and combinations will be within the spirit of the invention and the scope of the following claims.
This application is a continuation of U.S. application Ser. No. 16/209,854, titled “INTEGRATED PROGRAMMING FRAMEWORK FOR SPEECH AND TEXT UNDERSTANDING WITH BLOCK AND STATEMENT STRUCTURE”, filed Dec. 4, 2018, which is a continuation of Ser. No. 13/843,290, titled “INTEGRATED PROGRAMMING FRAMEWORK FOR SPEECH AND TEXT UNDERSTANDING WITH BLOCK AND STATEMENT STRUCTURE”, filed 15 Mar. 2013, which is related to and claims the benefit of U.S. Provisional Application No. 61/798,526 filed on Mar. 15, 2013, entitled “AN INTEGRATED PROGRAMMING FRAMEWORK FOR SPEECH AND TEXT UNDERSTANDING”, and U.S. Provisional Application No. 61/674,833 filed on Jul. 23, 2012, entitled “Terrier: An Integrated Programming Framework for Speech and Text Understanding”. The provisional applications are hereby incorporated by reference. This application is further related to U.S. application Ser. No. 13/480,400 filed on May 24, 2012, now U.S. Pat. No. 8,694,537, issued Apr. 8, 2014, entitled “SYSTEMS AND METHODS FOR ENABLING NATURAL LANGUAGE PROCESSING”, which is hereby incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3919479 | Moon et al. | Nov 1975 | A |
| 4450531 | Kenyon et al. | May 1984 | A |
| 4697209 | Kiewit et al. | Sep 1987 | A |
| 4739398 | Thomas et al. | Apr 1988 | A |
| 4843562 | Kenyon et al. | Jun 1989 | A |
| 4918730 | Schulze | Apr 1990 | A |
| 4928249 | Vermesse | May 1990 | A |
| 4959850 | Marui | Sep 1990 | A |
| 5019899 | Boles et al. | May 1991 | A |
| 5033087 | Bahl et al. | Jul 1991 | A |
| 5054074 | Bakis | Oct 1991 | A |
| 5164915 | Blyth | Nov 1992 | A |
| 5436653 | Ellis et al. | Jul 1995 | A |
| 5437050 | Lamb et al. | Jul 1995 | A |
| 5457768 | Tsuboi et al. | Oct 1995 | A |
| 5511000 | Kaloi et al. | Apr 1996 | A |
| 5542138 | Williams et al. | Aug 1996 | A |
| 5577249 | Califano | Nov 1996 | A |
| 5581658 | O'Hagan et al. | Dec 1996 | A |
| 5634084 | Malsheen et al. | May 1997 | A |
| 5664270 | Bell et al. | Sep 1997 | A |
| 5687279 | Matthews | Nov 1997 | A |
| 5708477 | Forbes et al. | Jan 1998 | A |
| 5845306 | Schabes et al. | Dec 1998 | A |
| 5862260 | Rhoads | Jan 1999 | A |
| 5874686 | Ghias et al. | Feb 1999 | A |
| 5880386 | Wachi et al. | Mar 1999 | A |
| 5907815 | Grimm et al. | May 1999 | A |
| 5918223 | Blum et al. | Jun 1999 | A |
| 5956683 | Jacobs et al. | Sep 1999 | A |
| 5963957 | Hoffberg | Oct 1999 | A |
| 5969283 | Looney et al. | Oct 1999 | A |
| 5974409 | Sanu et al. | Oct 1999 | A |
| 5991737 | Chen | Nov 1999 | A |
| 6049710 | Nilsson | Apr 2000 | A |
| 6067516 | Levay et al. | May 2000 | A |
| 6092039 | Zingher | Jul 2000 | A |
| 6098042 | Huynh | Aug 2000 | A |
| 6108626 | Cellario et al. | Aug 2000 | A |
| 6121530 | Sonoda | Sep 2000 | A |
| 6122403 | Rhoads | Sep 2000 | A |
| 6163767 | Tang et al. | Dec 2000 | A |
| 6182128 | Kelkar et al. | Jan 2001 | B1 |
| 6188985 | Thrift et al. | Feb 2001 | B1 |
| 6201176 | Yourlo | Mar 2001 | B1 |
| 6233682 | Fritsch | May 2001 | B1 |
| 6246986 | Ammicht et al. | Jun 2001 | B1 |
| 6292767 | Jackson et al. | Sep 2001 | B1 |
| 6314577 | Pocock | Nov 2001 | B1 |
| 6345256 | Milsted et al. | Feb 2002 | B1 |
| 6363349 | Urs et al. | Mar 2002 | B1 |
| 6385434 | Chuprun et al. | May 2002 | B1 |
| 6405029 | Nilsson | Jun 2002 | B1 |
| 6408272 | White et al. | Jun 2002 | B1 |
| 6434520 | Kanevsky et al. | Aug 2002 | B1 |
| 6453252 | Laroche | Sep 2002 | B1 |
| 6487532 | Schoofs et al. | Nov 2002 | B1 |
| 6504089 | Negishi et al. | Jan 2003 | B1 |
| 6505160 | Levy et al. | Jan 2003 | B1 |
| 6507727 | Rick | Jan 2003 | B1 |
| 6510325 | Mack, II et al. | Jan 2003 | B1 |
| 6519564 | Hoffberg et al. | Feb 2003 | B1 |
| 6535849 | Pakhomov et al. | Mar 2003 | B1 |
| 6542869 | Foote | Apr 2003 | B1 |
| 6594628 | Jacobs et al. | Jul 2003 | B1 |
| 6611607 | Davis et al. | Aug 2003 | B1 |
| 6614914 | Rhoads et al. | Sep 2003 | B1 |
| 6629066 | Jackson et al. | Sep 2003 | B1 |
| 6631346 | Karaorman et al. | Oct 2003 | B1 |
| 6633845 | Logan et al. | Oct 2003 | B1 |
| 6633846 | Bennett et al. | Oct 2003 | B1 |
| 6640306 | Tone et al. | Oct 2003 | B1 |
| 6708150 | Hirayama et al. | Mar 2004 | B1 |
| 6804645 | Kleinschmidt | Oct 2004 | B1 |
| 6834308 | Ikezoye et al. | Dec 2004 | B1 |
| 6850288 | Kurokawa | Feb 2005 | B2 |
| 6879950 | Mackie et al. | Apr 2005 | B1 |
| 6931451 | Logan et al. | Aug 2005 | B1 |
| 6941275 | Swierczek | Sep 2005 | B1 |
| 6967275 | Ozick | Nov 2005 | B2 |
| 6990453 | Wang et al. | Jan 2006 | B2 |
| 6995309 | Samadani et al. | Feb 2006 | B2 |
| 6996529 | Minnis | Feb 2006 | B1 |
| 7017208 | Weismiller et al. | Mar 2006 | B2 |
| 7058376 | Logan et al. | Jun 2006 | B2 |
| 7085716 | Even et al. | Aug 2006 | B1 |
| 7089541 | Ungar | Aug 2006 | B2 |
| 7174293 | Kenyon et al. | Feb 2007 | B2 |
| 7190971 | Kawamoto | Mar 2007 | B1 |
| 7206820 | Rhoads et al. | Apr 2007 | B1 |
| 7209892 | Galuten et al. | Apr 2007 | B1 |
| 7225132 | Attwater et al. | May 2007 | B2 |
| 7233321 | Larson et al. | Jun 2007 | B1 |
| 7257536 | Finley et al. | Aug 2007 | B1 |
| 7266343 | Yli-juuti et al. | Sep 2007 | B1 |
| 7266495 | Beaufays et al. | Sep 2007 | B1 |
| 7323629 | Somani et al. | Jan 2008 | B2 |
| 7328153 | Wells et al. | Feb 2008 | B2 |
| 7373209 | Tagawa et al. | May 2008 | B2 |
| 7379875 | Burges et al. | May 2008 | B2 |
| 7444353 | Chen et al. | Oct 2008 | B1 |
| 7464065 | Gleason et al. | Dec 2008 | B2 |
| 7516074 | Bilobrov | Apr 2009 | B2 |
| 7562392 | Rhoads et al. | Jul 2009 | B1 |
| 7567899 | Bogdanov | Jul 2009 | B2 |
| 7580832 | Allamanche et al. | Aug 2009 | B2 |
| 7606708 | Hwang | Oct 2009 | B2 |
| 7672916 | Poliner et al. | Mar 2010 | B2 |
| 7693720 | Kennewick et al. | Apr 2010 | B2 |
| 7698136 | Nguyen et al. | Apr 2010 | B1 |
| 7725319 | Aronowitz | May 2010 | B2 |
| 7743092 | Wood | Jun 2010 | B2 |
| 7756874 | Hoekman et al. | Jul 2010 | B2 |
| 7765097 | Yu et al. | Jul 2010 | B1 |
| 7783489 | Kenyon et al. | Aug 2010 | B2 |
| 7853664 | Wang et al. | Dec 2010 | B1 |
| 7858868 | Kemp et al. | Dec 2010 | B2 |
| 7881657 | Wang et al. | Feb 2011 | B2 |
| 7899818 | Stonehocker et al. | Mar 2011 | B2 |
| 7904297 | Mirkovic et al. | Mar 2011 | B2 |
| 7908135 | Shishido | Mar 2011 | B2 |
| 8013230 | Eggink | Sep 2011 | B2 |
| 8073684 | Sundareson | Dec 2011 | B2 |
| 8086171 | Wang et al. | Dec 2011 | B2 |
| 8099281 | Gleason | Jan 2012 | B2 |
| 8116746 | Lu et al. | Feb 2012 | B2 |
| 8296179 | Rennison | Oct 2012 | B1 |
| 8358966 | Zito et al. | Jan 2013 | B2 |
| 8447608 | Chang et al. | May 2013 | B1 |
| 8694537 | Mohajer | Apr 2014 | B2 |
| 8762156 | Chen | Jun 2014 | B2 |
| 8843369 | Sharifi | Sep 2014 | B1 |
| 8924212 | Allauzen et al. | Dec 2014 | B1 |
| 9646628 | Carlson et al. | May 2017 | B1 |
| 9697827 | Lilly et al. | Jul 2017 | B1 |
| 10224030 | Kiss et al. | Mar 2019 | B1 |
| 20010005823 | Fischer et al. | Jun 2001 | A1 |
| 20010014891 | Hoffert et al. | Aug 2001 | A1 |
| 20010049601 | Kroeker et al. | Dec 2001 | A1 |
| 20010053974 | Lucke et al. | Dec 2001 | A1 |
| 20020023020 | Kenyon et al. | Feb 2002 | A1 |
| 20020042707 | Zhao et al. | Apr 2002 | A1 |
| 20020048350 | Phillips et al. | Apr 2002 | A1 |
| 20020049037 | Christensen et al. | Apr 2002 | A1 |
| 20020072982 | Barton et al. | Jun 2002 | A1 |
| 20020083060 | Wang et al. | Jun 2002 | A1 |
| 20020111806 | Franz et al. | Aug 2002 | A1 |
| 20020116191 | Olsen et al. | Aug 2002 | A1 |
| 20020138265 | Stevens et al. | Sep 2002 | A1 |
| 20020138630 | Solomon et al. | Sep 2002 | A1 |
| 20020156627 | Itoh et al. | Oct 2002 | A1 |
| 20020174431 | Bowman et al. | Nov 2002 | A1 |
| 20020181671 | Logan | Dec 2002 | A1 |
| 20020193895 | Qian et al. | Dec 2002 | A1 |
| 20020198705 | Burnett | Dec 2002 | A1 |
| 20020198713 | Franz et al. | Dec 2002 | A1 |
| 20020198719 | Gergic et al. | Dec 2002 | A1 |
| 20020198789 | Waldman | Dec 2002 | A1 |
| 20030004717 | Strom et al. | Jan 2003 | A1 |
| 20030009335 | Schalkwyk et al. | Jan 2003 | A1 |
| 20030023437 | Fung | Jan 2003 | A1 |
| 20030040901 | Wang | Feb 2003 | A1 |
| 20030050784 | Hoffberg et al. | Mar 2003 | A1 |
| 20030078928 | Dorosario et al. | Apr 2003 | A1 |
| 20030083863 | Ringger et al. | May 2003 | A1 |
| 20030106413 | Samadani et al. | Jun 2003 | A1 |
| 20030110035 | Thong et al. | Jun 2003 | A1 |
| 20030125945 | Doyle | Jul 2003 | A1 |
| 20030163320 | Yamazaki et al. | Aug 2003 | A1 |
| 20030167166 | Mann | Sep 2003 | A1 |
| 20030187649 | Logan et al. | Oct 2003 | A1 |
| 20030191645 | Zhou | Oct 2003 | A1 |
| 20030192424 | Koike | Oct 2003 | A1 |
| 20030233225 | Bond | Dec 2003 | A1 |
| 20040002858 | Attias et al. | Jan 2004 | A1 |
| 20040019497 | Volk et al. | Jan 2004 | A1 |
| 20040167779 | Lucke et al. | Aug 2004 | A1 |
| 20040193420 | Kennewick et al. | Sep 2004 | A1 |
| 20040231498 | Li et al. | Nov 2004 | A1 |
| 20050010412 | Aronowitz | Jan 2005 | A1 |
| 20050016360 | Zhang | Jan 2005 | A1 |
| 20050027699 | Awadallah et al. | Feb 2005 | A1 |
| 20050033574 | Kim et al. | Feb 2005 | A1 |
| 20050060685 | Franz et al. | Mar 2005 | A1 |
| 20050086059 | Bennett | Apr 2005 | A1 |
| 20050125232 | Gadd | Jun 2005 | A1 |
| 20050137939 | Calabria et al. | Jun 2005 | A1 |
| 20050143970 | Roth et al. | Jun 2005 | A1 |
| 20050144064 | Calabria et al. | Jun 2005 | A1 |
| 20050144065 | Calabria et al. | Jun 2005 | A1 |
| 20050254366 | Amar | Nov 2005 | A1 |
| 20050256715 | Okimoto et al. | Nov 2005 | A1 |
| 20050273326 | Padhi et al. | Dec 2005 | A1 |
| 20060004572 | Ju et al. | Jan 2006 | A1 |
| 20060059225 | Stonehocker et al. | Mar 2006 | A1 |
| 20060069547 | Wang et al. | Mar 2006 | A1 |
| 20060095250 | Chen et al. | May 2006 | A1 |
| 20060122839 | Li-Chun Wang et al. | Jun 2006 | A1 |
| 20060129396 | Ju et al. | Jun 2006 | A1 |
| 20060155694 | Chowdhury et al. | Jul 2006 | A1 |
| 20060169126 | Ishiwata et al. | Aug 2006 | A1 |
| 20060189298 | Marcelli | Aug 2006 | A1 |
| 20060200350 | Attwater et al. | Sep 2006 | A1 |
| 20060224384 | Dow et al. | Oct 2006 | A1 |
| 20060242017 | Libes et al. | Oct 2006 | A1 |
| 20060277030 | Bedworth | Dec 2006 | A1 |
| 20060277052 | He et al. | Dec 2006 | A1 |
| 20060282266 | Lopez-Barquilla et al. | Dec 2006 | A1 |
| 20070010195 | Brown et al. | Jan 2007 | A1 |
| 20070016404 | Kim et al. | Jan 2007 | A1 |
| 20070038453 | Yamamoto et al. | Feb 2007 | A1 |
| 20070055500 | Bilobrov | Mar 2007 | A1 |
| 20070120689 | Zerhusen et al. | May 2007 | A1 |
| 20070156392 | Balchandran et al. | Jul 2007 | A1 |
| 20070168409 | Cheung | Jul 2007 | A1 |
| 20070168413 | Barletta et al. | Jul 2007 | A1 |
| 20070204319 | Ahmad et al. | Aug 2007 | A1 |
| 20070208569 | Subramanian et al. | Sep 2007 | A1 |
| 20070239676 | Stonehocker et al. | Oct 2007 | A1 |
| 20070260456 | Proux et al. | Nov 2007 | A1 |
| 20070260634 | Makela et al. | Nov 2007 | A1 |
| 20070288444 | Nelken et al. | Dec 2007 | A1 |
| 20080022844 | Poliner et al. | Jan 2008 | A1 |
| 20080026355 | Petef | Jan 2008 | A1 |
| 20080046247 | Kurata et al. | Feb 2008 | A1 |
| 20080059185 | Chung et al. | Mar 2008 | A1 |
| 20080059188 | Konopka et al. | Mar 2008 | A1 |
| 20080071520 | Sanford | Mar 2008 | A1 |
| 20080082510 | Wang et al. | Apr 2008 | A1 |
| 20080126089 | Printz et al. | May 2008 | A1 |
| 20080134264 | Narendra et al. | Jun 2008 | A1 |
| 20080148224 | Donovan et al. | Jun 2008 | A1 |
| 20080154951 | Martinez et al. | Jun 2008 | A1 |
| 20080172224 | Liu et al. | Jul 2008 | A1 |
| 20080215319 | Lu et al. | Sep 2008 | A1 |
| 20080228496 | Yu et al. | Sep 2008 | A1 |
| 20080235017 | Satomura | Sep 2008 | A1 |
| 20080235872 | Newkirk et al. | Oct 2008 | A1 |
| 20080249982 | Lakowske | Oct 2008 | A1 |
| 20080255937 | Chang et al. | Oct 2008 | A1 |
| 20080256115 | Beletski et al. | Oct 2008 | A1 |
| 20080270129 | Colibro et al. | Oct 2008 | A1 |
| 20090013255 | Yuschik et al. | Jan 2009 | A1 |
| 20090030686 | Weng et al. | Jan 2009 | A1 |
| 20090031882 | Kemp et al. | Feb 2009 | A1 |
| 20090037174 | Seltzer et al. | Feb 2009 | A1 |
| 20090063147 | Roy | Mar 2009 | A1 |
| 20090064029 | Corkran et al. | Mar 2009 | A1 |
| 20090112593 | Konig et al. | Apr 2009 | A1 |
| 20090119097 | Master et al. | May 2009 | A1 |
| 20090125298 | Master et al. | May 2009 | A1 |
| 20090125301 | Master et al. | May 2009 | A1 |
| 20090125306 | Lejeune et al. | May 2009 | A1 |
| 20090150341 | Paek et al. | Jun 2009 | A1 |
| 20090216525 | Shostak | Aug 2009 | A1 |
| 20090228799 | Verbeeck et al. | Sep 2009 | A1 |
| 20090240488 | White et al. | Sep 2009 | A1 |
| 20090240499 | Dvir et al. | Sep 2009 | A1 |
| 20090271199 | Agapi et al. | Oct 2009 | A1 |
| 20100014828 | Sandstrom et al. | Jan 2010 | A1 |
| 20100042414 | Lewis et al. | Feb 2010 | A1 |
| 20100049514 | Kennewick et al. | Feb 2010 | A1 |
| 20100100384 | Ju et al. | Apr 2010 | A1 |
| 20100124892 | Issa et al. | May 2010 | A1 |
| 20100158488 | Roberts et al. | Jun 2010 | A1 |
| 20100205166 | Boulter et al. | Aug 2010 | A1 |
| 20100211693 | Master et al. | Aug 2010 | A1 |
| 20100235341 | Bennett | Sep 2010 | A1 |
| 20100241418 | Maeda et al. | Sep 2010 | A1 |
| 20100250497 | Redlich et al. | Sep 2010 | A1 |
| 20100286979 | Zangvil et al. | Nov 2010 | A1 |
| 20100312782 | Li et al. | Dec 2010 | A1 |
| 20110035219 | Kadirkamanathan et al. | Feb 2011 | A1 |
| 20110046951 | Suendermann et al. | Feb 2011 | A1 |
| 20110071819 | Miller et al. | Mar 2011 | A1 |
| 20110082688 | Kim et al. | Apr 2011 | A1 |
| 20110131043 | Adachi et al. | Jun 2011 | A1 |
| 20110132173 | Shishido | Jun 2011 | A1 |
| 20110132174 | Shishido | Jun 2011 | A1 |
| 20110173208 | Vogel | Jul 2011 | A1 |
| 20110213475 | Herberger et al. | Sep 2011 | A1 |
| 20110244784 | Wang | Oct 2011 | A1 |
| 20110276334 | Wang et al. | Nov 2011 | A1 |
| 20110288855 | Roy | Nov 2011 | A1 |
| 20120029670 | Mont-Reynaud et al. | Feb 2012 | A1 |
| 20120035924 | Jitkoff et al. | Feb 2012 | A1 |
| 20120046936 | Kandekar et al. | Feb 2012 | A1 |
| 20120065960 | Iwama et al. | Mar 2012 | A1 |
| 20120089400 | Henton | Apr 2012 | A1 |
| 20120216178 | Gellerich et al. | Aug 2012 | A1 |
| 20120232683 | Master et al. | Sep 2012 | A1 |
| 20120303371 | Labsky et al. | Nov 2012 | A1 |
| 20120323557 | Koll et al. | Dec 2012 | A1 |
| 20130006631 | Gunther et al. | Jan 2013 | A1 |
| 20130041647 | Ramerth et al. | Feb 2013 | A1 |
| 20130052939 | Anniballi et al. | Feb 2013 | A1 |
| 20130055223 | Xu | Feb 2013 | A1 |
| 20130096911 | Beaufort et al. | Apr 2013 | A1 |
| 20130111440 | Forster | May 2013 | A1 |
| 20130151250 | VanBlon | Jun 2013 | A1 |
| 20140019483 | Mohajer | Jan 2014 | A1 |
| 20140032220 | Lerner | Jan 2014 | A1 |
| 20140039895 | Aravamudan et al. | Feb 2014 | A1 |
| 20140067394 | Abuzeina et al. | Mar 2014 | A1 |
| 20140074470 | Jansche et al. | Mar 2014 | A1 |
| 20140205974 | Pellom et al. | Jul 2014 | A1 |
| 20140297252 | Prasad et al. | Oct 2014 | A1 |
| 20140316785 | Bennett et al. | Oct 2014 | A1 |
| 20140324427 | Di Fabbrizio et al. | Oct 2014 | A1 |
| 20140324433 | Hsiao | Oct 2014 | A1 |
| 20140358533 | Kurata et al. | Dec 2014 | A1 |
| 20150039317 | Klein et al. | Feb 2015 | A1 |
| 20150106082 | Ge et al. | Apr 2015 | A1 |
| 20150112679 | Zhang | Apr 2015 | A1 |
| 20150161985 | Peng et al. | Jun 2015 | A1 |
| 20160148615 | Lee et al. | May 2016 | A1 |
| 20160232894 | Park et al. | Aug 2016 | A1 |
| 20170178623 | Shamir et al. | Jun 2017 | A1 |
| 20180061399 | Rose et al. | Mar 2018 | A1 |
| 20180308489 | Pan et al. | Oct 2018 | A1 |
| 20180330723 | Acero et al. | Nov 2018 | A1 |
| 20220374708 | Rozenfarb | Nov 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 0944033 | Sep 1999 | EP |
| 1367590 | Dec 2003 | EP |
| H11-272274 | Oct 1999 | JP |
| 2000187671 | Jul 2000 | JP |
| 1517746 | Jun 1995 | WO |
| 9918518 | Apr 1999 | WO |
| 03061285 | Jul 2003 | WO |
| 2004091307 | Oct 2004 | WO |
| 2008004181 | Jan 2008 | WO |
| 2010018586 | Feb 2010 | WO |
| 2013177213 | Nov 2013 | WO |
| Entry |
|---|
| Murase, T, et al., “Incremental CFG Parsing with Statistical Lexical Dependencies”, Proceedings of the Sixth Natural Language Processing Pacific Rim Symposium, Nov. 2001, 9 pgs. |
| New, B., “Question Answering at TREC”, Pre-Internship report, Mar. 2008. 4 pgs. |
| Nortel Norstar, “Voice Mail Speech Recognition Automated Attendant”, Product manual [online]. 29 pgs. Nortel Norstar [retrieved Sep. 4, 2012]. Retrieved from the Internet: <URL: https://downloads.avaya.com/css/P8/documents/100141923>. |
| Norvell, T., “A Short Introduction to Regular Expressions and Context Free Grammars”, Project report, Nov. 2002, 5 pgs. |
| Quesada, J, et al. “Design of a Natural Language Command Dialogue System”. Project deliverable 3.2, SIRIDUS, 2000, 91 pgs. |
| Seneff, S., “TINA: A Natural Language System for Spoken Language Applications”, Journal of the Association for Computational Linguistics, 18 (1), pp. 61-82, Mar. 1992. |
| Seneff, S, et al., “Galaxy—II: A Reference Architecture for Conversational System Development”, Proceedings of the International Conference on Spoken Language Processing, Nov. 1998, 5 pgs. |
| Stolcke, A., “An Efficient Probabilistic Context-Free Parsing Algorithm that Computes Prefix Probabilities”, Journal of the Association for Computational Linguistics, 21 (2), Jun. 1995, pp. 165-201. |
| “Do you know the true cost of IVR migration?” Datasheet. Aspect Software Incorporated, Dec. 2012, 4 pgs. |
| Zlatanov, T. “Cultured Perl”, Nov. 2004, 13 pgs. [retrieved Oct. 22, 2014]. Retrieved from the Internet: <URL: http://www.ibm.com/developerworks/linux/library/l-cpregex/index.html>. |
| Younger, D. H., “Recognition and parsing of context-free languages in time n3”, Information and Control, vol. 10, Issue 2, Feb. 1967, pp. 189-208. |
| Guzzino, Didier, “Active: A Unified Platform for Building Intelligent Applications”, Jan. 23, 2008, 263 pgs. |
| “Nuance Recognizer 9.0: Language Pack Guide”, Nuance Communications, Inc., 2007, 34 pgs. |
| U.S. Appl. No. 13/842,735—Office Action dated Feb. 20, 2015, 12 pages. |
| U.S. Appl. No. 13/480,400—Office Action dated May 7, 2013, 15 pages. |
| U.S. Appl. No. 13/480,400—Response to May 7 Office Action filed Jul. 8, 2013, 16 pages. |
| U.S. Appl. No. 13/843,290—Notice of Allowance dated Jan. 23, 2015, 9 pages. |
| U.S. Appl. No. 13/844,028—Office Action dated Nov. 7, 2014, 15 pages. |
| PCT/US2009/066458—International Search Report, dated Jun. 23, 2010, 16 pages. |
| InData Corporation, DepoView Video Review Software Product Description, “InData's Newest Video Deposition Viewer”, Dec. 2007, 2 pgs. |
| InData Corporation, DepoView DVD, Video Review Software Product Brochure, Jun. 2008, 4 Pgs. |
| InData Corporation, DepoView Video Review Software Product Description, http://indatacorp.com/depoview.html, accessed Nov. 8, 2011, 2 Pgs. |
| Sony Ericsson's W850i Walkman Phone Now Available in the Middle East. Al-Bawaba News, 2006 Al-Bawaba. Dec. 11, 2006. Factiva, Inc. <www.albawaba.com>. 2 pages. |
| Blackburn, S G., “Content Based Retrieval and Navigation of Music,” University of Southampton, Department of Electronics and Computer Science, Faculty of Engineering and Applied Science, Mar. 10, 1999, 41 Pages. |
| Blackburn, S., et al. “A Tool for Content Based Navigation of Music,” University of Southampton, Department of Electronics and Computer Science, Multimedia Research Group, Copyright 1998 ACM 1-58113-036-8/98/0008, pp. 361-368. |
| Blackburn, S G., “Content Based Retrieval and Navigation of Music Using Melodic Pitch Contours,” University of Southampton, Department of Electronics and Computer Science, Faculty of Engineering and Applied Science, Sep. 26, 2000, 136 Pages. |
| Blackburn, S G. “Search by Humming”. University of Southampton, Department of Electronics and Computer Science, Faculty of Engineering, May 8, 1997, 69 Pages. |
| Hum That Tune, Then Find it on the Web. NPR: Weekend Edition—Saturday, WKSA. Copyright 2006 National Public Radio. Dec. 23, 2006. Factiva, Inc. 2 pages. |
| Casey, M. A., et al., “Content-Based Music Information Retrieval: Current Directions and Future Challenges”. Apr. 2008, vol. 96, No. 4, Copyright 2008 IEEE, Retrieved from IEEE Xplore [retrieved on Dec. 29, 2008 at 18:02], 29 Pages. |
| Wagstaff, J., “Loose Wire: New Service Identifies Songs You Hum,” WSJA Weekend Journal. Copyright 2006, Dow Jones & Company, Inc. Dec. 25, 2006. Factiva, Inc. 2 pages. |
| Saltzman, M., “The Best Things in life are Free—For Your iPhone,” Home Electronics and Technology, for Canwest News Service. Copyright 2008 Edmonton Journal. Nov. 12, 2008. Factiva, Inc. 2 pages. |
| First Products with Gracenote Technology to Ship in 2008. Warren's Consumer Electronics Daily. Copyright 2007 Warren Publishing, Inc. Sep. 18, 2007. Factiva, Inc. 2 pages. |
| Gracenote Readies New Services, But Video Initiative Stalls. Warren's Consumer Electronics Daily. Copyright 2005 Warren Publishing, Inc. vol. 5; Issue 122. Jun. 24, 2005. Factiva, Inc. 2 pages. |
| Furui, S., “Digital Speech Processing, Synthesis, and Recognition”. Second Edition, Revised and Expanded. Nov. 17, 2000. ISBN 978-0824704520. 17 pages. |
| Ghias, A., et al. “Query by Humming,” Musical Information Retrieval in an Audio Database, Cornell University 1995, 6 Pages. |
| Ghias, A., et al. “Query by Humming—Musical Information Retrieval in an Audio Database,” ACM Multimedia 95—Electronic Proceedings, San Francisco, CA, Nov. 5-9, 1995, 13 Pages. |
| Han, B., et al. “M-Musics: Mobile Content-Based Music Retrieval System”. Copyright 2007, Augsburg, Bavaria, Germany. ACM 978-1-59593-01-8/07/0009. Sep. 23-28, 2007. pp. 469-470. 2 Pages. |
| Jang, J.R., et al. “A General Framework of Progressive Filtering and its Application to Query to Singing/Humming”. IEEE Transactions on Audio, Speech, and Language Processing, vol. 16. No. 2, Feb. 2008. pp. 350-358. 9 Pages. |
| Kosugi, N., et al. “A Practical Query-By-Humming System for a Large Music Database”. NTT Laboratories, Japan. ACM Multimedia Los Angeles, Ca, USA. Copyright ACM 2000 1-58113-198-4/00/10. pp. 333-342. 10 Pages. |
| McNab, R. J., et al. “Towards the Digital Music Library: Tune Retrieval from Acoustic Input”. University of Waikato, Department of Computer Science, School of Education. DL 1996, Bethesda MD USA. Copyright 1996 ACM 0-89791-830-4/96/03. pp. 11-18. 8 Pages. |
| McNab, R. J., et al. “The New Zealand Digital Library MELody inDEX”. University of Waikato, Department of Computer Science. D-Lib Magazine, May 1997 [retrieved on Jun. 12, 2011 at 11:25:49 AM]. ISSN 1082-9873. Retrieved from the Internet: <http://dlib.org/dlib/May97/meldex/05wiillen.html>, 13 pages. |
| Pardo, B., et al. “The VocalSearch Music Search Engine”. EECS, Northwestern University. JCDL 2008, Pittsburgh, Pennsylvania, USA. Jun. 16-20, 2008, ACM 978-1-59593-998-2/08/06. p. 430. 1 Page. |
| Mobile Music: Comcast Cellular First in U.S. to Trial Breakthrough Interactive Music Service Called *CD. Copyright PR Newswire, New York. ProQuest LLC. Feb. 11, 1999. Retrieved from the Internet: <http://proquest.umi.com.libproxy.mit.edu/pqdwb?did+38884944&sid=3&Fmt=3&clientId=5482&RQT=309&VName=PQD>. 3 pages. |
| Song, J., et al. “Query by Humming: Matching Humming Query to Polyphonic Audio,” LG Electronics, Seoul, Korea. Copyright 2002 IEEE. 0-7809-7304-9/02. pp. 329-332. 4 Pages. |
| Taylor, C., “Company Lets Listeners Dial for CDs,” Billboard, vol. 1, No. 26, General Interest Module, Jun. 26, 1999, pp. 86-87, 2 pages. |
| “Can't Get That Song Out of Your Head,” Copyright 2007, The Jakarta Post, May 20, 2007, Factiva, Inc, 2 Pages. |
| Typke, R., et al., “A Survey of Music Information Retrieval Systems,” Universiteit Utrecht, The Netherlands. Copyright 2005 Queen Mary, University of London. 8 Pages. |
| Wang, A., “The Shazam Music Recognition Service”. Communications of the ACM, vol. 49, No. 8. Aug. 2006. ACM 0001-0782/06/0800. pp. 44-48. 5 pages. |
| Melodis Rolls Out midomi mobile. Wireless News. Copyright 2008 M2 Communications, Ltd. Mar. 6, 2008. 1 Page. |
| Zhu, Y., et al. “Warping Indexes with Envelope Transforms for Query by Humming”. New York University, New York. SIGMOD Copyright 2003, San Diego, CA. Jun. 9-12, 2003. ACM 1-58113-634-X/03/06. pp. 181-192. 12 Pages. |
| U.S. Appl. No. 13/843,290—Notice of Allowance dated Aug. 15, 2016, 5 pages. |
| U.S. Appl. No. 13/843,290—Response to Notice of Allowance dated Aug. 15, 2016, filed Nov. 15, 2016, 11 pages. |
| U.S. Appl. No. 13/842,735—Final Office Action dated May 22, 2017, 10 pages. |
| U.S. Appl. No. 13/843,290—Notice of Allowance dated Sep. 5, 2017, 7 pages. |
| De Mareail, Philippe Boula, et al. “A French Phonetic Lexicon with Variants for Speech and Language Processing.” LREC. Jun. 2000, pp. 1-4. |
| Perennou, Guy, et al. “MHATLex: Lexical Resources for Modelling the French Pronunciation.” LREC. Jun. 2000, pp. 1-8. |
| Ranbom, Larissa J., et al. “Lexical representation of phonological variation in spoken word recognition.” Journal of Memory and Language 57.2, Aug. 2007, pp. 273-298. |
| Svendsen, Torbjorn. “Pronunciation modeling for speech technology.” Proc. Intl. Conf. on Signal Processing and Communication (SPCOM04), Jan. 2005, pp. 1-6. |
| Young, S. L., et al. “High level knowledge sources in usable speech recognition systems.” Communications of the ACM 32.2, Feb. 1989, pp. 183-194. |
| Gaudinat, Amaud, et al. “Syntax-based speech recognition: how a syntactic parser can help a recognition system.” EuroSpeech. Sep. 1999, pp. 1-4. |
| U.S. Appl. No. 14/634,642—Non-final Office Action dated Sep. 22, 2017, 24 pages. |
| U.S. Appl. No. 13/842,735—Response to Final Office Action dated May 22, 2017, filed Nov. 21, 2017, 11 pages. |
| U.S. Appl. No. 13/842,735—Nonfinal Office Action dated Dec. 15, 2017, 14 pages. |
| U.S. Appl. No. 13/843,290—Response after Notice of Allowance dated Sep. 5, 2017, filed Nov. 27, 2017, 10 pages. |
| U.S. Appl. No. 13/843,290—Notice of Allowance dated Jan. 31, 2018, 5 pages. |
| U.S. Appl. No. 13/193,514—Response to Aug. 22, 2014 Office Action, filed Dec. 22, 2014, 11 pages. |
| U.S. Appl. No. 14/634,642—Notice of Allowance dated Mar. 7, 2018, 22 pages. |
| Yao, “The Effects of Phonological Neighborhoods in Pronunication Variation in Conversational Speech”, UC Berkeley Electroinic Theses and Dissertatioms, Jan. 1, 2011, 218 pages. |
| U.S. Appl. No. 14/634,642—Response to Non-final Office Action dated Sep. 22, 2017 filed Jan. 22, 2018, 29 pages. |
| U.S. Appl. No. 14/634,642—Supplemental Response to Non-final Office Action dated Sep. 22, 2017 filed Jan. 22, 2018, 15 pages. |
| U.S. Appl. No. 14/634,642—Interview Summary and Proposed Examiner's Amendment filed Jan. 30, 2018, 10 pages. |
| U.S. Appl. No. 14/634,642—Amendment with RCE filed Jun. 6, 2018, 14 pages. |
| U.S. Appl. No. 14/634,642—Non-final Office Action dated Jun. 27, 2018, 14 pages. |
| U.S. Appl. No. 14/634,642—Response to Non-final Office Action dated Jun. 27, 2018 filed Dec. 27, 2018, 9 pages. |
| U.S. Appl. No. 14/634,642—Final Office Action dated Feb. 12, 2019, 12 pages. |
| Huet, et al., “Morpho-syntactic post-processing of N-best lists for improved French automatic speech recognition”, Computer Speech and Langauge; ScienceDirect; 22 pages; received Jul. 9, 2008; available online Oct. 30, 2009. |
| U.S. Appl. No. 13/843,290—Response after Notice of Allowance dated Jan. 18, 2018, filed Apr. 25, 2018, 10 pages. |
| U.S. Appl. No. 13/843,290—Notice of Allowance dated Sep. 5, 2018, 11 pages. |
| U.S. Appl. No. 13/842,735—Response to Office Action dated 125 Jun. 2019, filed Nov. 25, 2019, 11 pages. |
| U.S. Appl. No. 13/842,735—Final Office Action dated Dec. 13, 2019, 14 pages. |
| PCT/US2009/066458—International Preliminary Report on Patentability dated Jun. 7, 2011, 7 pages. |
| U.S. Appl. No. 60/222,023—Wang, et al., “Method and Apparatus for Recognizing Sound and Music Signals in High Noise and Distortion” filed Jul. 31, 2000, 26 pages. |
| U.S. Appl. No. 60/134,782—Rhoads, G., “Methods and Systems Employing Digital Watermarking,” filed May 19, 1999, 47 pages. |
| U.S. Appl. No. 60/166,965—Finley, M., et al., “Broadcast Media Purchasing System,” filed Nov. 23, 1999, 21 pages. |
| U.S. Appl. No. 60/158,087—Swierczek, R., “Music Identification System,” filed Oct. 7, 1999, 12 pages. |
| U.S. Appl. No. 60/186,565—Swierczek, R., “Music Identification System,” filed Mar. 2, 2000, 14 pages. |
| Chou, T., et al., “Music Databases: Indexing Techniques and Implementation”, Proceedings of International Workshop on Multimedia Database Management Systems, IEEE, dated Aug. 14-16, 1996, pp. 46-53, 8 pages. |
| McPherson, J.R., et al., “Usage of the MELDEX Digital Music Library”, 1999, in Proceedings of the International Symposium on Music Information Retrieval, (Bloomington, IN, USA, 2001), pp. 19-20, 2 pages. |
| Wold, E., et al., “Classification, Search, and Retrieval of Audio”, Muslce Fish, Berkeley, CA, USA, CRC Handbook of Multimedia Computing 1999, pp. 1-19, 18 pages. |
| Wold, et al., “Content-Based Classification, Search and Retrieval of Audio”, IEEE Multimedia 1070-986X/96, vol. 3, No. 3: Fall 1996, pp. 27-36 (17 pages). |
| Horn, P., “What was that song? With a wireless phone, find out what you heard on the radio,” The Inquirer, Philadelphia, Pennsylvania, USA, dated Feb. 11, 1999, 3 pages. |
| U.S. Appl. No. 60/218,824—Kenyon, S., et al., “Audio Identification System and Method,” filed Jul. 18, 2000, 45 pages. |
| U.S. Appl. No. 60/155,064—Kenyon, S., “Automatic Program Identification System and Method,” filed Sep. 21, 1999, 49 pages. |
| U.S. Appl. No. 13/401,728—Notice of Allowance dated Mar. 4, 2015, 8 pages. |
| U.S. Appl. No. 13/401,728—Office Action dated Jul. 17, 2014, 11 pages. |
| U.S. Appl. No. 13/193,514—Office Action dated Jul. 17, 2015, 16 pages. |
| U.S. Appl. No. 13/193,514—Office Action dated Aug. 22, 2014, 21 pages. |
| U.S. Appl. No. 13/193,514—Office Action dated Jan. 6, 2014, 21 pages. |
| U.S. Appl. No. 13/193,514—Response to Jan. 6 Office Action filed May 6, 2014, 11 pages. |
| Amdal, et al., “Pronunciation variation modeling in automatic speech recogniction”, Teletronikk 99.2, Feb. 2003, pp. 70-82. |
| Bisani, et al., “Joint-sequence models for grapheme-to-phoneme conversion”, Speech Communication 50.5, May 2008, pp. 434-451. |
| Harrison, et al., “Improving mispronunciation detection and diagnosis of learners' speech with context-sensitive phonological rules bases on language transfer”, INTERSPEECH, Sep. 2008, pp. 2787-2790. |
| Tesprasit, et al., “A context-sensitive homograph disambiguation in Thai text-to-speech synthesis”, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: companion volume of hte Proceedings of HLT-NAACL 2003, (2003), pp. 1-3. |
| U.S. Appl. No. 14/634,642—Office Action dated Mar. 23, 2016, 15 pages. |
| Phillips, W., “Introduction to Natural Language Processing,” CCSI 2006 The Mind Project, retrieved online: <http://www.mind.ilstu.edu/curriculum/protothinker/natural_language_processing.php> accessed on Nov. 25, 2016, 24 pages. |
| Feldman, “NLP Meets the Jabberwocky: Natural Language Processing in Information Retrieval,” Online, Information Today, Inc., Copyright May 1999, 16 pages. |
| Wilcox, B., “Beyond Façade: Pattern Matching for Natural Language Applications,” The Gamasutra—The Art & Business of Making Games, Copyright 2016 UBM Tech, retrieved online: <http://gamasutra.com/view/feature/134765/beyond_fa%C3%A7ade_pattern_matching_.php> accessed on Nov. 25, 2016, 14 pages. |
| U.S. Appl. No. 13/842,735—Response to Feb. 20 Office Action filed Aug. 19, 2015, 15 pages. |
| U.S. Appl. No. 13/842,735—Office Action dated Nov. 10, 2016, 8 pages. |
| Bechet, F., et al., “Large Span statistical models: application to homophone disambiguation for large vocabulary speech recognition in French,” Eurospeech, Sep. 1999, p. 1-4. |
| Choi, J., et al., “On applying phonological rules for a., Korean Continuous Speech Recognition System,” Proceedings of ICSP '97, Aug. 1997, pp. 491-496. |
| Cremelie, et al., “In search of better pronounciation models for speech recognition,” Speech Communication 29.2, Nov. 1999, pp. 115-136. |
| Derouault, A., et al., “Natural language modeling for phoneme-to-text transcription,” IEEE Transactions on Pattern Analysis & Machine Intelligence, Nov. 6, 1986, pp. 742-749. |
| Gravier, G., et al., “Introducing contextual transcription rules in large vocabulary speech recognition,” The Integration of Phonetic knowledge in speech technology, Spinger Netherlands, 2005, pp. 87-106. |
| Holter, T., et al., “Maximum likelihood modelling of pronounciation variation,” Speech Communication 29.2, Nov. 1999, pp. 177-191. |
| Miller, L.G., et al., “Syntactic Analysis for large vocabulary speech recognition using a context-free covering grammar,” Acoustics, Speech, and Signal processing, 1988, ICASSP—88, 1988 International Conference on. IEEE, Apr. 1988, pp. 271-274. |
| Schaden, S., “Rule-based lexical modelling of foreign-accented pronounciation variants,” Proceedings of the tenth conference on European chapter of the Association of Computation Linguistics—vol. 2, Association for Computational Linguistics, 2003, pp. 159-162. |
| Schiel, F., et al., “Statistical modeling of pronunciation: it's not the model, it's the data,” Modeling Pronounciation Variation for Automatic Speech Recognition, May 1998, pp. 1-7. |
| U.S. Appl. No. 14/634,642—Response to Mar. 23 Office Action filed May 17, 2016, 13 pages. |
| U.S. Appl. No. 14/634,642—Office Action dated Jul. 28, 2016, 27 pages. |
| U.S. Appl. No. 14/634,642—Response to Jul. 28 Office Action filed Sep. 15, 2016, 8 pages. |
| U.S. Appl. No. 13/843,290—Office Action dated Jan. 19, 2017, 7 pages. |
| U.S. Appl. No. 13/842,735—Notice of Allowance dated Jul. 13, 2016, 10 pages. |
| U.S. Appl. No. 13/842,735—Response to Office Action dated Nov. 10, 2016, filed May 8, 2017, 11 pages. |
| U.S. Appl. No. 13/842,735—Notice of Allowance dated Sep. 11, 2015, 10 pages. |
| U.S. Appl. No. 13/842,735—Notice of Allowance dated Jan. 7, 2016, 11 pages. |
| U.S. Appl. No. 13/842,735—Response to Notice of Allowance dated Jul. 13, 2016, filed Oct. 11, 2016, 14 pages. |
| U.S. Appl. No. 13/843,290—Notice of Allowance dated May 18, 2015, 5 pages. |
| U.S. Appl. No. 13/843,290—Notice of Allowance dated Aug. 3, 2015, 5 pages. |
| U.S. Appl. No. 13/843,290—Notice of Allowance dated Feb. 5, 2016, 5 pages. |
| Ranta, A., “Grammatical Framework: Programming with Multilingual Grammars,” Slides for the GF book, CSLI Stanford, Copyright 2011, 440 pages. |
| Ranta, A., “Grammatical Framework Tutorial”, Copyright Dec. 2010 for GF 3.2 [retrieved on Sep. 13, 2012], Retrieved from Internet: <http://www.grammaticalframework.org/doc/tutorial/gf-tutorial.html>. 68 pages. |
| “Grammatical Framework” Version 3.3.3, Copyright Mar. 2012 [retrieved on Sep. 23, 2012], Retrieved from Internet: <http://www.grammaticalframework.org>. 4 pages. |
| VoiceXML Tutorial (NUANCE), BeVocal, Inc., Mountain View, CA, Copyright 2005, 68 pages. |
| JavaScript Quick Reference, BeVocal Inc. (NUANCE), Mountain View, CA, Copyright 2005, 24 pages. |
| Grammar Reference, BeVocal, Inc. (NUANCE), Mountain View, CA, Copyright 2005, 102 pages. |
| Ranta, A., “Creating Linguistic Resources with the Grammatical Framewor,” LREC Tutorial, Malta, May 17, 2010, 75 pages. |
| Conway, D., et al., “Synopsis 5: Regexes and Rules”, Version 158, Created Jun. 24, 2002 and Last Modified Jul. 31, 2012 [retrieved Sep. 26, 2012], Retrieved from Internet: <http://perlcabal.org/syn/s05.html>, 65 pages. |
| Grammar's Developers Guide, Nuance Speech Recognition System, Version 8.5, Copyright 2003 Nuance Communications Inc., Menlo Park, CA, 262 pages. |
| Wang, A.L., “An Industrial-Strength Audio Search Algorithm,” In ISMIR 2003, 4th Symposium Conference on Music Information Retrieval(Oct. 26, 2003), pp. 7-13. |
| Venkatachalam, V., Cazzanti, L., Chillon, N., Wells, M., “Automatic Identification of Sound Recordings,” Signal Processing Magazine, IEEE, Mar. 2004, 92-99, vol. 21, Issue 2. |
| Nelson, J., V Cast Song ID from Verizon Wireless. May 21, 2007 [retrieved on Jul. 24, 2014], Retrieved from Internet: <http://www.verizonwireless.com/news/article/2007/05/pr2007-05-21a.html>. |
| Gracenote: MusicID, available at http://www.gracenote.com/business.sub.--solutions/music.sub.--id/, last accessed Aug. 4, 2010. |
| Shazam: http://web.archive.org/web/20100501190631/http://www.shazam.com/. Last accessed May 1, 2010. |
| App Shopper Shazam: http://appshopper.com/music/shazam. Last changed Jul. 24, 2014. |
| Gracenote Mobile MusicID: http://web.archive.org/web/20100123211802/http://www.gracenote.com/busine- ss.sub.-solutions/mobileMusic/. Last accessed Jan. 23, 2010. |
| App Shopper MusicID: http://appshopper.com/music/musicid. Last changed Jul. 14, 2014. |
| Xu, et al. “Music Identification via Vocabulary Tree with MFCC Peaks,” MIRUM '11 Proceedings of the 1st international ACM workshop on Music information retrieval with user-centered and multimodal strategies, 2011. p. 21-26.http://dl.acm.org/citation.cfm?doid=2072529.2072537. |
| Li, et al. “Robust Audio Identification for MP3 Popular Music,” SIGIR '10 Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, Jul. 2010. p. 627-634.http://dLacm.org/citation.cfm?doid=1835449.1835554. |
| Yu, et al. “A Query-by-Singing System for Retrieving Karaoke Music,” IEEE Transactions on Multimedia, Dec. 2008, vol. 10, No. 8, p. 1626-1637. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4694852. |
| Liu, et al. “Content-Based Retrieval of MP3 Music Objects,” CIKM '01 Proceedings of the tenth international conference on Information and knowledge management, 2001. p. 506-511. http://dx.doi.org/10.1145/502585.502670. |
| OMRAS2—Ontology-Driven Music Retrieval & Annotation Sharing Service. Overview—Apr. 24, 2009 [Accessed Jul. 24, 2014—Archive.org] http://web.archive.org/web/20090424083019/http://www.omras2.org/overview. 2 pages. |
| OMRAS2—AudioDB-Populating and Querying an AudioDB Instance. (No archived version available—accessed Jul. 24, 2014 via Google) http://omras2.org/audioDB/tutorial1. 3 pages. |
| Benson, et al. “Sync Kit: A Persistent Client-Side Database Caching Toolkit for Data Intensive Websites,” Proceedings of the 19th International Conference on Worid Wide Web, Apr. 2010. pp. 121-130. http://dl.acm.org/citation.cfm?id=1772704. cited byapplicant. |
| Larson, et al. “NYT to Release Thesaurus and Enter Linked Data Cloud,” NY Times Blogs, Jun. 2009. http://open.blogs.nytimes.com/2009/06/26/nyt-to-release-thesaurus-and-ent- erlinked-data-cloud/. |
| “Aurix Enhances Effectiveness of Leading Search Software,” Aurix.com—News. Jun. 1, 2010. http://www.aurix.com/pages/3808/Aurix.sub.-enhances.sub.--effectiveness.- sub.-of .sub.--leading.sub.-search.sub.-software.htm. |
| “Hearing it Loud & Clear at SpeechTEK 2010,” Aurix.com—News. Jul. 21, 2010, http://www.aurix.com/pages/4161/State.sub.-of .sub.--the.sub.--art.sub.--speech.sub.-technology.htm. |
| Jamil, “A Natural Language Interface Plug-In for Cooperative Query Answering in Biological Databases,” BMC Genomics, Nov. 2011. (Accessed Sep. 27, 2012 via SpringerLink) http://www.biomedcentral.com/1471-2164/13/S3/S4. |
| Feng, “A General Framework for Building Natural Language Understanding Modules in Voice Search,” 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), Mar. 2010. (Accessed Jul. 24, 2014—IEEE)http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5494951. |
| Langanke, “Direct Voice Control Speech Data Entry and Database Query Models,” International Symposium on Logistics and Industrial Informatics, Sep. 2007. (Accessed Jul. 24, 2014—IEEE)http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4343522. |
| Indukuri, et al. “Natural Language Querying Over Databases Using Cascaded CRFs,” Lecture Notes in Computer Science, Sep. 2010, http://www.springerlink.com/content/5w1x27650475304m. |
| Kolias, et al. “Design and implementation of a VoiceXML-driven wiki application for assistive environments on the web,” Personal and Ubiquitous Computing, Sep. 2010. vol. 14, No. 6, p. 527-539,http://www.icsd.aegean.gr/publication.sub.-files/journal/295233664.pdf. |
| PCT/US2013/042097—International Search Report & Written Opinion dated Dec. 2, 2013. |
| ATT, “Conversant VIS Version 4.0 Whole Word Bilignual Speech Recognition”, Issue 1, Oct. 1993, 86 pgs. |
| “Contact Center Business Planning Guide”, Avaya Inc., 2010, 8 pgs. |
| “Avaya Self-Service Optimization. Optimize the Performace of your Avaya Self-Service applications”, Avaya, Inc., 2011, 4 pgs. |
| “VoiceXML Tutorial. Developer documentation”, Bevocal, Inc., 2005, 68 pgs. |
| Brick, T, et al. “Incremental Natural Language Parsing for HRI”, Journal of the Association for Computing Machinery, Mar. 10-12, 2007, 8 pgs. |
| Charniak, E., et al. “Edge-Based Best-First Chart Parsing”, 1998, 8 pgs. |
| Crisostomo, A. “Constituents and Phrases”, Jul. 2011, 43 pgs. |
| Copestake, A, et al., “Minimal Recursion Semantics: An Introduction” Research on Language and Computation, vol. 3, pp. 281-332, 2005. |
| Deobhakta, N., “Natural Language Processing, User Interfaces, and the Semantic Web”. Proceedings from the Third Annual HCI Symposium, Dec. 2012, 22 pgs. |
| “ECMAScript Language Specification”, ECMA-262, ECMA International, 5.1 Edition, Jun. 2011, 258 pgs. |
| Graham, P., “Parsing with ATNs. On Lisp: Advanced Techniques for Common Lisp”, Engelwood Cliffs, NJ, Prentice Hall, 1993, 16 pgs. |
| McKeown, K., “Semantic Analysis: Syntax-Driven Semantics”, 27 pgs. [retrieved Jun. 17, 2014]. Retrieved from the Internet: <URL: <http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CCEQFjAB&url=http%3A%2F%2Fwww1.cs.columbia.edu%2F˜kathy%2FNLP%2FClassSlides%2FClass13SemanticAnalysis%2Fsemantics.ppt&ei=Xdb4VMPDIcvsoAS2soGABg&usg=AFQjCNGiXuBWLO-oQB_MLor_kN_8ATdpRg&sig2=BnvJvvJJo3OApAC6ny0guQ&bvm=bv.87611401,d.cGU>>. |
| Huang, L, et al., “Dynamic Programming for Linear-Time Incremental Parsing”. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 1077-1086, Jul. 2010. |
| Iqbal, R, et al., “A Negation Query Engine for Complex Query Transformations”. Journal of Science and Technology, pp. 193-204, 2013. |
| Klein, D, et al. “Accurate Unlexicalized Parsing”, Proceedings of the 41st Meeting of the Association for Computational Linguistics, vol. 1, pp. 423-430, 2003. |
| Matsubara, S, et al., “Chart-based Parsing and Transfer in Incremental Spoken Language Translation”, Proceedings of the Fourth Natural Language Processing Pacific Rim Symposium, 1997, 4 pgs. |
| Mohri, M, et al., “Weighted Finite-State Transducers in Speech Recognition”, Computer Speech and Language, Jan. 2002, 26 pgs. |
| Jurafsky et al., Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition, 3rd edition draft dated Sep. 21, 2021, 629 pages. Retrieved on Nov. 4, 2021. Retrieved from the internet [URL: https://web.stanford.edu/˜jurafsky/slp3/ed3book_sep212021.pdf ]. |
| Number | Date | Country | |
|---|---|---|---|
| 20210224043 A1 | Jul 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61798526 | Mar 2013 | US | |
| 61674833 | Jul 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16209854 | Dec 2018 | US |
| Child | 17225997 | US | |
| Parent | 13843290 | Mar 2013 | US |
| Child | 16209854 | US |
