Patent Application
20190188255
The present disclosure relates generally to generating correction candidates for spell checking, and is applicable to generating correction candidates in a morphological-rich language in which words are derived from a finite set of morphological patterns, in particular the Arabic language.
Generating suitable correction candidates has become an ever increasingly required feature in any spell checking and correction system. Correction candidates' generators generally calculate a set of scores describing the similarity between an error word and the possible corrections. The most commonly used approach for generating correction candidates in Arabic is employing the edit distance. Other approaches generate correction candidates based on specific words. A further approach has been to perform analysis of error detection and correction at the sentence level.
The foregoing “Background” description is for the purpose of generally presenting the context of the disclosure. Work of the inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present invention.
Bakkali, et. al. proposed the use of a small size dictionary of stems to represent Arabic to help correcting spelling errors, instead of using a large dictionary. (See Bakkali, H., et al. “For an Independent Spell-Checking System from the Arabic Language Vocabulary.” International Journal of Advanced Computer Science and Applications 5 (2014), herein incorporated by reference in its entirety). This dictionary is similar to the one used by Buckwalter Aramorph morphological analyzer. For every input word that is not in the dictionary, a distance measure between the morphemes of the analyzed word and the Aramorph lexicon tables is computed and the suggested correction is determined based on the minimum distances. Using a corpus of 2784 misspelled words, the authors reported that their method outperformed the classical Levenshtein approach in terms of the average time (0.10/0.19 ms) and the correction rate (85%/50%).
Nejja, et al. used the concept of surface patterns and roots with the Levenshtein minimum edit distance to generate corrections of the error word. (See Nejja, Mohammed, and Yousfi Abdellah. “Correction of the Arabic derived words using surface patterns.” Codes, Cryptography and Communication Systems (WCCCS), 2014 5th Workshop on. IEEE, 2014), herein incorporated by reference in its entirety), herein incorporated by reference in its entirety). Nejja, et al. provides three approaches for the correction process. These approaches are mainly based on selecting the surface pattern that has minimum edit distance with the correct word. The approaches were compared to the Levenshtein edit distance. The performance of the system is reportedly better than that of the edit distance in terms of the execution time and the lexicon size.
A context-based system was suggested to automatically correct misspelled words by Alkanhal, et al. (See Alkanhal, Mohamed, et al. “Automatic stochastic arabic spelling correction with emphasis on space insertions and deletions.” Audio, Speech, and Language Processing, IEEE Transactions on 20.7 (2012): 2111-2122), herein incorporated by reference in its entirety). The misspelt words are firstly ranked using the Levenshtein edit distance considering space insertion and space deletion errors. The most correct candidate for each misspelled word is then selected according to the maximum marginal probability via A* lattice search and N-gram probability estimation. They reported an improved performance reaching F-scores of 97.9% and 92.3% for detection and correction, respectively, based on their manually annotated corpora.
Hassan, Y. et. al., proposed a system that targets the detection and correction of several error types of QALB shared task corpus, including edit, add before, merge and split errors. (See Hassan, Y., Aly, M., & Atiya, A. (2014). Arabic Spelling Correction using Supervised Learning. arXiv preprint arXiv: 1409.8309), herein incorporated by reference in its entirety). The system detects erroneous words by applying Buckwalter morphological analyzer. For each detected “Edit” and “add before” errors, classifiers with contextual features are used to correct them. A random insertion and omitting of a space were maintained to correct merge and split errors. They reported 58% F1 error correction score with 59% and 58% recall and precision, respectively.
Hicham introduced an approach for correcting errors that result from insertion, deletion and substitution operations. (See Hicham, Gueddah. “Introduction of the weight edition errors in the Levenshtein distance.” arXiv preprint arXiv:1208.4503 (2012)), herein incorporated by reference in its entirety). When ranking candidates, Hicham used the frequency of change operations with the Levenshtein distance. This approach achieved better performance than using Levenshtein edit distance alone.
Shaalan, et al., 2012 created a large-coverage word list for Arabic of 13 million words, with 9 million having fully inflected valid surface words using AraComLex, an open-source finite-state large-scale morphological transducer. (See Shaalan, Khaled F., et al. “Arabic Word Generation and Modelling for Spell Checking.” LREC. 2012), herein incorporated by reference in its entirety). From a large list of valid forms and invalid forms, he created a character-based tri-gram language model to approximate knowledge about permissible character clusters in Arabic, creating a novel method for detecting spelling errors. He also created a context-independent spelling correction tool using a finite-state automaton that measures the edit distance between input words and candidate corrections. He matched the word against the list to detect whether it was an error word or not. If it is an error word, he used finite-state automata to propose candidate corrections within a specified edit distance measured by the Levenshtein distance from the misspelled word. After choosing candidate corrections, he used the noisy channel model trained on a corpus of one-billion words, and knowledge-based rules to assign scores to the candidate corrections and choose the best correction, independent of the context. The knowledge-based rules focused on the most common types of errors.
Mahdi implemented a spell checker that uses morphological analysis for the detection of non-word spelling errors. (See Mahdi, Adnan. 2012. “Spell Checking and Correction for Arabic Text Recognition.” Master's thesis, KFUPM University, Department of Information & Computer Science, herein incorporated by reference in its entirety). The word identified as spelling error if a table look-up fails to match the generated morpheme. Once an error is detected, suggested corrections are generated based on Levenshtein distance. No test data or results were discussed in this work.
Attia, et al. developed knowledge-based re-ranking rules for reordering and constraining the number of candidates generated though the Levenshtein edit distance. (See Attia, Mohammed, et al. “Improved Spelling Error Detection and Correction for Arabic.” COLING (Posters). 2012), herein incorporated by reference in its entirety). He showed that careful selection of the language model training data based on the amount of noise present in the data, has the potential to further improve the overall results. Their method significantly outperforms the three industrial applications of Ayaspell, MS Word, and Google Docs in first order ranking of candidates.
Hassan, A., et al. proposed a language independent method for detecting and correcting spelling errors. (See Hassan, Ahmed, Sara Noeman, and Hany Hassan. “Language Independent Text Correction using Finite State Automata.” IJCNLP. 2008), herein incorporated by reference in its entirety). Error detection is based on finite state automata while candidates' generation was handled using a Levenshtein-transducer that is compatible with the finite state machine. Ranking candidates was handled using the n-gram language model. Hassan, A, et al. reported an accuracy of 89% based on Arabic and English text.
Shaalan, et al., 2003, developed an Arabic morphological analyzer that is able to recognize common spelling errors for standard Arabic and Egyptian language dialects. (See Shaalan, Khaled, Amin Allam, and Abdallah Gomah. “Towards automatic spell checking for Arabic.” Conference on Language Engineering, ELSE, Cairo, Egypt. 2003), herein incorporated by reference in its entirety). Shaalan, et al., 2003 used a heuristic approach that attempts to find the correction of the error. Shaalan, et al., 2003 built the system using prolog.
A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:
Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout several views, the following description relates to a spell checking error model for generating correction candidates for a particular error word.
The Arabic language is considered to be one of the morphologically rich languages, since many of its words are derived from a finite set of morphological patterns. Arabic is considered to be one of the Semitic languages, which are known for their morpheme diversity. For example, in Arabic, every derived word follows a specific morphological pattern that centers around, usually a 3-letter root word, along with any of the allowable prefixes, suffixes and inflictions, if any. Root words that have more than 3 letters are rare. Subsequently, Arabic morphology assigns the pattern to each such root word, where the first letter corresponds to
, the second letter corresponds to
, and the third letter corresponds to
. In addition, not every prefix and/or suffix generates proper derived Arabic words.
As mentioned above, various approaches to implementing spell checkers generally focus on analyzing a word. The inventors for the present disclosure have determined that morphological patterns in words can be utilized in generating smarter candidate words for correcting spelling errors. Such a morphological pattern-based model preferably possesses the following properties:
Disclosed is a data driven system that exploits morphological error patterns at morphemes or word levels. Although examples are disclosed that are based on the Arabic language, the disclosure applies to other languages that are morphologically rich, such as other Semitic languages. An aspect is an error pattern that is associated with a single morphological pattern, which may be applied to any specific word.
The ECPD 105 and CCG 107 for generating correction candidates 130 for a given error word 120 may be incorporated into a variety of systems. For example the disclosed data driven system may be incorporated as a spell checking feature in a word processing program, or may be incorporated as part of a grammar checker. The disclosed data driven system may be incorporated as part of a voice recognition system where speech input is displayed as text output. In the later example, the source of the error word is a computer generated word derived from the speech input.
Error Correction Patterns Generator (ECPG)
Regarding,
The flowchart in
Regarding
’ in the context of ‘
’, the correction word ‘
’ is analyzed using SWAM, and as a result is identified as ‘
’ with morphological pattern ‘
’. The morphological pattern and the real word error are used to generate an error pattern ‘
’, as shown in
Regarding
The error encoding EE which is generated in this example is a string that specifies the positions of the changes and the change types in the error pattern. In the example, there are two changes: transposition of the first two characters of the surface pattern (BC in CP Label) and a substitution of the first character in the suffix (′). Hence, the resulting error encoding (EE) is ‘-s$-BCS--’, according to the error types listed in
An example of the steps in the EE and CC generation process are shown in
In S317, the character in CP is compared to the character in iEP. If a comparison between the character in CP and the character in iEP indicates that there is No Change (YES in S317): in S319, error code ‘-’ is added to Error Encoding (EE). If the CP character is not a label but an affix position character (‘S’) (YES in S321), in S323, the affix position character (‘$’) is added to both EE and CC. If the character in CP is a label (YES in S325), and the character in iEP is not a label (NO in S327), there is letter insertion: in S329, error code ‘i’ is added to EE. If there is letter deletion (YES in S331): in S333, ‘d’ is added to EE; in S335, the deleted letter is added to Correction Code (CC). If there is letter substitution (NO in S331): in S337, ‘s’ is added to EE; in S335, the substituted letter is added to CC. If there is letter transposition (YES in S327): in SS341, the character in CP label is added to CC; in S343, the correct label is added to EE. In S311, the final EP is extracted according to EE and iEP. The results of this step are shown in “Step 5” in
Error Correct Patterns Database (ECPD)
The generated error patterns, error encodings and correction codes are stored in the ECPD 105.
Correct Candidates' Generator (CCG)
Regarding ’) match their corresponding letters in the error word. In S909, matching patterns are generated. In S911, matching patterns are evaluated to determine if they have a valid root. Correction candidates that have a valid root are added to a list of valid candidates. The CCG 107 continues until all error patterns are considered.
An example is provided as shown in ’, the CCG 107 matches all the morphological error patterns, similar in length to the input, against the input error word and generates all the correction candidates according to the matched patterns correction information. In
’) match their corresponding letters in the error word. Non-valid candidates are automatically rejected as possible candidates; a correction (Correction) is valid (Valid=True) if it has a valid root (Root). According to the example database in
’ provides valid candidates.
In one implementation, the functions and processes of the system of
Further, the claimed advancements may be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with CPU 1100 and an operating system such as Microsoft® Windows®, UNIX®, Oracle® Solaris, LINUX®, Apple macOS® and other systems known to those skilled in the art.
In order to achieve the computer 1126, the hardware elements may be realized by various circuitry elements, known to those skilled in the art. For example, CPU 900 may be a Xenon® or Core® processor from Intel Corporation of America or an Opteron® processor from AMD of America, or may be other processor types that would be recognized by one of ordinary skill in the art. Alternatively, the CPU 1100 may be implemented on an FPGA, ASIC, PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize. Further, CPU 900 may be implemented as multiple processors cooperatively working in parallel to perform the instructions of the inventive processes described above.
The computer 1126 in
The computer 1126 further includes a display controller 1108, such as a NVIDIA® GeForce® GTX or Quadro® graphics adaptor from NVIDIA Corporation of America for interfacing with display 1110, such as a Hewlett Packard® HPL2445w LCD monitor. A general purpose I/O interface 1112 interfaces with a keyboard and/or mouse 1114 as well as an optional touch screen panel 1116 on or separate from display 1110. General purpose I/O interface also connects to a variety of peripherals 1118 including printers and scanners, such as an OfficeJet® or DeskJet® from Hewlett Packard®.
The general purpose storage controller 1120 connects the storage medium disk 904 with communication bus 1122, which may be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the computer 1126. A description of the general features and functionality of the display 1110, keyboard and/or mouse 1114, as well as the display controller 1108, storage controller 1120, network controller 906, and general purpose I/O interface 1112 is omitted herein for brevity as these features are known.
System Analysis
To study the effectiveness of the suggested error model, an analysis of some well-known Arabic annotated corpora was conducted. The analysis is concentrated on the types and number of edits needed to get the correct word from the error word. The corpus chosen for this task is that of QALB. QALB is a manually annotated errors corpus consisting of user comments collected from the Al-Jazeera News webpage. They are, mainly, written in Modern Standard Arabic. It is mainly written in Modern Standard Arabic with almost one Million words with 243,075 errors. QALB has its specific format. However, its format was converted into KACST format. The corpus files are then fed to the morphological analyzer (SWAM) to get the morphological pattern of the corpus words. The corpus has different types of errors, some of errors are annotated with more than three words as correction. Only sentences that have, among their errors, less than four connected words were included; the reason behind this exclusion is that such errors are fairly unique and they introduce overhead over the morphological analyzer disambiguation process.
Table 1 shows the summary of the QALB corpus errors based on the error model analysis result file. The deletion error type ratio (%40.18) in Table 1 includes the percentage of punctuation deletion which is %33.45.
Most of the errors in QALB are repeated errors, the total number of distinct corpus errors is 89536 as shown in Table 2. The total number of distinct errors consists of uniquely occurred errors (Non-Repeated error words, 68253) and the distinct number of repeated error words (21283).
The summary statistics of the QALB corpus error patterns based on the error model analysis result file are shown in Table 3Table.
The total number of distinct generated error patterns is 46129 as shown in Table 4. The generated total distinct error patterns are composed of uniquely generated patterns (Non-Repeated error patterns, 68253) and the distinct number of repeated error patterns (21283).
With the fact that the total number of the analyzed corpus errors is 243075; the statistics show that almost 71% (243075−68253=174822) of these errors are just a repetition of only 21283 distinct errors with an average of almost 8 occurrence per error word. The remaining 29% of the corpus errors are singly occurred errors (non-repeated errors). This means that any upcoming error word should have their correction candidate in the list (if a simple non-error patterns model is used) with the probability of at least 71%.
Although, the number of uniquely occurred errors in the corpus is 68253 which is almost triple the number of distinct repeated errors; the total number of the whole generated error patterns by the error model is 46129 (51.52% of the whole distinct corpus errors).
We conducted an analysis of the actual error model candidates' generation effectiveness compared to the Levenshtein minimum edit distance based on QALB test corpus. The experiment concentrated on the ability of the error model, based on the learnt error patterns, to generate the correct candidates among all the suggested candidates. The ranking of candidates was handled based on the error pattern repetition frequency. Different training corpora were used to train the error model. For example, Table 5 shows that, in case of using QALB training data, 83% of the correct correction does exist in the generated correction candidates. The average number of candidates is 20 and the average candidates' generation time is 0.11. Moreover, almost 0.17% of the corrections are exist among the top correction.
In the case of Levenshtein minimum edit distance, the candidates were generated using an edit distance of 1 and 2. The candidates with the required edit distance are generated from a dictionary1 with 125975 words. Candidates' generation within the edit distance of 3 was not included since the generation time was very large (almost 33 minutes/error word) and the memory requirements were huge (4800 candidates/error word).
Numerous modifications and variations are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.
Thus, the foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.
This application claims the benefit of priority to provisional application No. 62/599,396 filed Dec. 15, 2017, the entire contents of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62599396 | Dec 2017 | US |
