Patent Application
20050153267
1. Field of the Invention
This invention relates in general to the field of computer based training to improve neurological function in humans.
2. Description of the Related Art
The present application will describe a computer based software program entitled “Fast ForWord”, developed by Scientific Learning Corporation. Information about this program can be found at http://www.scientificlearning.com. Dr. Michael M. Merzenich of the present invention was a co-inventor of that program, for which a number of patents have been granted. The present application utilizes a portion of the description of the Fast ForWord program found in U.S. Pat. No. 5,927,988 entitled “METHOD AND APPARATUS FOR TRAINING OF SENSORY AND PERCEPTUAL SYSTEMS IN LLI SUBJECTS” which is hereby incorporated by reference for all purposes. The present invention has made numerous improvements to the Fast ForWord program to obtain results which are the subject of the present application. Before these improvements are described, however, a brief overview of current research on the neurology of aging is provided.
Overview of Current Research Emphasis on the Neurology of Aging
Scientific views about the neurological bases of the loss of function in aging ultimately terminating in Alzheimer's Disease (AD) or in other forms of dementia have been dominated by a focus on a large body of evidence, summarized by tens of thousands of research reports, documenting the physical deterioration of the brain in aging. Aging neurons become dysfunctional and die because of a pathological overgrowth of their microtubules, because emergent amyloid or Lewy body accretions set processes in motion that disable and kill them, and because of a variety of other documented factors that parallel or can lead to neuronal deterioration and loss. The interconnections between neurons become progressively sparser and less complex. Neuron processes (spines, dendrites, axons) supporting those interconnections progressively simplify. The basal metabolism of key brain structures, the production of critical neurotransmitters, and other important processes enabling normal memory and learning functions are progressively down-regulated. The neuronal regulation of cortical neurovascular responses gradually weakens, and degraded blood perfusion control exacerbates pathological aging processes.
These (and other) chemical, physical and physiological changes in aging brains documented at autopsy or through brain imaging or brain response recording have been repeatedly correlated with impairments in memory, cognition, motor control, mood control, and other brain/behavioral processes and abilities. For example, the fewer the numbers of surviving cortical neurons or the greater the physical or chemical indices of functional deterioration or loss in limbic system nuclei or in temporal or anterior cingulate cortex, the greater the cognitive impairments, and the poorer the immediate- and delayed-recall memory abilities. The greater the deterioration of the middle temporal lobe reflected by neuropil shrinkage or cell loss, the greater an individual's difficulties at word retrieval or naming. The greater the deterioration of the middle and inferior temporal cortex, the greater the predicted impairments in face recognition and complex visual memory or visual association. The greater the physical deterioration of the hippocampus, the greater the deficits in episodic memory. A number of other similar correlative arguments relating specific physical aspects of brain pathology (e.g., intracellular ‘tangles’, amyloid bodies, et alia) to behavioral impairments have been reported in the aging/AD literature.
Studies of the origins of cognitive impairments and AD symptoms have frequently focused on specific structures implicated as playing particularly important roles in memory, cognition or motor control. A large experimental and clinical literature has targeted the abnormal state of the hippocampus and entorhinal cortex in the aged infirm. Other studies have documented large-scale differences in the physical and functional status of the anterior cingulate or temporal cortices. Still others have documented functional and morphological changes in the basal nucleus of Meynert, and in related “modulatory control system” nuclei. Many other reports have documented basal ganglia and cortical changes that parallel cognitive deficits and a loss of motor control that can ultimately lead to another great plague for aged populations, Parkinsonism.
Collectively, this massive research literature establishes six well-established and unchallengeable principles: 1) Neurons and the richness of their interconnections (brain ‘neuropil’) are progressively lost and reduced as we age. 2) Emergent pathological processes that effectively ‘poison’ the brain contribute directly to that loss, and mark the progression from ‘normal aging’ to ‘Mild Cognitive Impairment’ (MCI) to Alzheimer's Disease (AD). 3) The deteriorating brain machinery includes nuclei and cortical areas that are specifically related to learning, memory, cognition, mood, and voluntary and involuntary movement control. 4) The metabolic decline and a down-regulation of the specific functions of key neuronal populations commonly precedes cell death. 5) These changes are inexorable. Although there is substantial variability in the times of onset, time courses, and magnitudes of functional and physical deterioration, they are a universal outcome of the later years of an extended human life. 6) A large number of dimensions of physical and chemical deterioration and of emergent neuropathology are correlated with general and specific behavioral losses.
What is needed is a novel set of computer-based training exercises, based on the established science of “brain plasticity”, consisting of separate but related training modules that, in aggregate, significantly improve fundamental aspects of brain performance and function relevant to the remediation of the neurological origins and consequences of age-related cognitive decline (ARCD).
Further, what is needed is a method and apparatus that induces an adult brain within an appropriate behavioral context to improve perceptual, cognitive, executive control, mood control, and motor skill development. More specifically what is needed is a method and apparatus to provide repetitive learning exercises which are modulated with surprises and rewards to achieve faster and stronger learning.
To address the above-detailed deficiencies, the present invention provides a method for providing normal and surprise rewards to a human during training.
In one aspect, the present invention provides a method for improving neurological processes in a human, the method employing visual and acoustic computer based training games. The method includes: providing one or more training games to the human, each of the one or more training games having a plurality of trials; presenting one of the plurality of trials from the one or more training games to the human, as a trial; determining whether the human correctly responded to the trial; if the human correctly responded to the trial, determining whether an increased reward should be presented; if an increased reward should be presented, presenting the increased reward; if an increased reward should not be presented, but the human correctly responded to the trial, presenting a normal reward. The increased reward is not presented simply because the human correctly responded to the trial.
In another aspect, the present invention provides a method on a computing device for effecting positive neurological function on a human, the method providing unexpected surprise rewards. The method includes: within a video/audio computing game context, presenting a series of trials to a human, each of the trials having at least one correct response and at least one incorrect response; when the human selects an incorrect response, not providing a reward; when the human selects a correct response, determining whether the human should be presented with a normal reward, or a surprise reward; and presenting the reward, whether normal or surprise; wherein the surprise reward is presented randomly according to a predetermined frequency.
In a further aspect, the present invention provides a method for rewarding correct selections in a computing game context designed to stimulate neurological development. The method includes: providing a normal reward for a correct selection; providing a surprise reward for a correct selection; and when a correct selection is made, occasionally presenting the surprise reward rather than the normal reward.
These and other objects, features, and advantages of the present invention will become better understood with regard to the following description, and accompanying drawings where:
Discussion of Program “Fast Forward”
Referring to
Now referring to
Referring to
Also shown are frequency components for a phoneme /ba/. This phoneme contains an upward sweep frequency component 308, at approximately 2 khz, having a duration of approximately 35 ms. The phoneme also contains an upward sweep frequency component 310, at approximately 1 khz, during the same 35 ms period. Following the stop consonant portion /b/ of the phoneme, is a constant frequency vowel portion 314 whose duration is approximately 110 ms.
Thus, both the /ba/ and /da/ phonemes begin with stop consonants having modulated frequency components of relatively short duration, followed by a constant frequency vowel component of longer duration. The distinction between the phonemes exist primarily in the 2 khz sweeps during the initial 35 ms interval. Similarity exists between other stop consonants such as /ta/, /pa/, /ka/ and /ga/.
Referring now to
With the above general background of speech elements, and how LLI subjects process them, a general overview of speech processing will now be provided. As mentioned above, one problem that exists in LLI subjects is the inability to distinguish between short duration acoustic events. If the duration of these acoustic events are stretched, in the time domain, it is possible to train LLI subjects to distinguish between these acoustic events. An example of such time domain stretching is shown in
In
Another method that may be used to help LLI subjects distinguish between phonemes is to emphasize selected frequency envelopes within a phoneme. Referring to
A third method that may be used to train LLI subjects to distinguish short duration acoustic events is to provide frequency sweeps of varying duration, separated by a predetermined interval, as shown in
Utilization of up-down frequency sweeps with varying ISI has been fully described in U.S. Pat. No. 5,813,862 entitled “METHOD AND DEVICE FOR ENHANCING THE RECOGNITION OF SPEECH AMONG SPEECH-IMPAIRED INDIVIDUALS”, and is hereby incorporated by reference.
Each of the above described methods have been combined in a unique fashion by the present invention to provide an adaptive training method and apparatus for training subjects having abnormal temporal processing abilities to recognize and distinguish short duration acoustic events that are common in speech. The present invention is embodied into a computer program entitled Fast ForWord by Scientific Learning Corporation. The computer program is provided to an LLI subject via a CD-ROM which is input into a general purpose computer such as that described above with reference to
Referring first to
When a subject begins execution of the Fast ForWord computer program, he/she is presented with a screen similar to the screen 800. More specifically, upon initiation of the program, the subject is presented with a screen that lists the subjects that are currently being trained by the program. The subject then selects his/her name from the list. Once the subject has selected his/her name, a screen similar to 800 appears, typically listing one of the seven programs, according to a training schedule that is dictated by the program, or is modified by an instructor. The order of the games, and the selection of which one of the seven games that is presented in the screen 800 varies from day to day. The subject then elects to play the first game listed according to the training schedule prescribed for the subject.
In one embodiment, a training schedule is provided by a certified Speech and Language Professional (SLP), and the SLP oversees each training session according to the schedule. An exemplary schedule requires a subject to cycle through five of the seven games for an hour and forty minutes, five days per week, for approximately six weeks. In addition, the schedule typically requires that a subject play Circus Sequence and Language Comprehension Builder everyday, alternating the other games so that they are played approximately the same amount of time.
In an alternative embodiment, the game schedule specified by an SLP at a remote server, and the daily parameters of the schedule are downloaded to the subject's computer, either daily or weekly. The schedule can be optimized over the course of the training program to first develop skills required for subsequent more advanced skills. It can also be used to help manage time in each game so that all of the games are completed at about the same time at the end of the training program. This embodiment allows a subject to obtain the benefits of the Fast ForWord program, and the oversight of a certified SLP, regardless of his/her geographic location. One skilled in the art will appreciate that the training schedule could either be provided in a window on the subject's computer, or could actually control the game selection screen to prompt the user only for those games required on a particular day.
Once a subject selects a particular game, he/she is taken into that particular game's module. Alternatively, once the subject selects his/her name from the list, the particular games may be presented, in a predefined order, without requiring the subject to first select the game. For ease of illustration, each of the seven games will be discussed, in the order represented in
Referring to
A number of scenes are provided in OMDFF, each correlated to a specific pair of sounds. The correlation of sound pairs to farm scenes is shown below:
So, when a subject grabs the flying animal, the game begins presenting a tone pattern such as: /Si/ . . . /Si/ . . . /Si/ . . . /Si/ . . . /Sti/. When the subject hears /Sti/, the subject is to release the animal.
The scene 900 provides a general farmyard background with three elements that persist across all the scenes. The elements are the score digits 906, the stop sign 908, and the tractor 910. The tractor 910 acts as a progress creature to graphically indicate to a subject their progress during a game. If the subject gets a correct response, the tractor 910 advances across the screen 900, from right to left. The score digits 906 display the subject's current score. The stop sign 908 is common to all seven games, and provides a subject with a means for exiting the game, and then the program.
Also shown on the screen 900 are a flying farm animal 902, and a selection hand 904. In this scene, the flying farm animal 902 is a cow with a rocket pack. Other scenes provide different farm animals propelled through the air with different flying apparatus. Operation of the game OMDFF will now be described with reference to
In
At block 1004, the computer program selects a particular tone sequence to be played for a subject. For example, the program would select the tone pair /Si/ . . . /Sti/, stretched 150%, with an ISI of 500 ms. The tone pair that is selected, the stretching, and the ISI, are all associated with a particular skill level. And, the skill level that is presented to a subject is adapted in real time, based on the subjects ability to recognize the target phoneme, as will be further described below. However, the initial phoneme pair, stretching and ISI are chosen to allow an LLI subject to understand the game, and to begin to distinguish phonemes common in speech. Upon selection of a particular phoneme sequence, and skill level, flow proceeds to block 1006.
At block 1006, the game presents a flying animal 902. As mentioned above, the animal 902 that is presented varies according to which of the phoneme pairs are selected. If the animal 902 is a flying cow, the phoneme pair that will be presented is /Gi/ . . . /Ki/. The animal 902 continues to fly around the screen until the subject places the selection hand 904 over the animal 902, and holds down a selection button, such as a mouse button. After the animal 902 is presented, flow proceeds to decision block 1008.
At decision block 1008, a test is made as to whether the subject has selected the animal 902. If not, flow proceeds to block 1010 where the animal 902 continues to fly. The animal 902 will continue moving about the scene 900 until it is selected. Flow then proceeds to block 1012.
At block 1012, the program begins presenting the selected phoneme sequence. More specifically, an audio formatted file is called by the program that is to be played by a computer, either through speakers connected to the computer, or though headphones worn by a subject. In one embodiment, the file is a QuickTime audio file, configured according to the parameters necessary for the skill level of the user, i.e., phoneme pair, stretching, and ISI. In addition, a starting point in the file is chosen such that the distractor phoneme is presented a random number of times, between 3 and 8 times, before the target phoneme is presented. After the phoneme sequence begins playing, flow proceeds to decision block 1014.
At decision block 1014, a determination is made as to whether the subject has released the animal 902. If the subject has not released the animal 902, a parallel test is made, shown as decision block 1016.
Decision block 1016 tests whether a “hit” window has passed. More specifically, the program contains a lockout window of 200 ms that begins when the target phoneme is played. It is believed that if the subject releases the animal 902 within 200 ms of the target phoneme beginning play, it is merely coincidental that he/she would have heard the target phoneme. This is because no subject's reaction time is quick enough release the animal 902 so soon after hearing the target phoneme. The start of the “hit” window begins after the lockout window, i.e., 200 ms after the target phoneme begins. The end of the hit window is calculated as the start of the hit window, plus the length of one phoneme letter. So, at decision block 1016, if the hit windows has not passed, the computer continues to test whether the subject has released the animal 902. If the hit window has passed, and the subject has not released the animal 902, flow proceeds to block 1026.
At block 1026, a miss is recorded for that test. After recording the miss, flow proceeds back to block 1021.
At block 1021, the skill level for the selected phoneme sequence is decreased, as will be further described below. Flow then proceeds back to block 1006 where another flying animal is presented for the same phoneme sequence.
At decision block 1014, if it is determined that the subject has released the animal 902, instruction flow proceeds to decision block 1018.
At decision block 1018, a determination is made as to whether the hit window has begun. That is, did the subject release the animal 902 during or before the lockout period? If the hit window has not begun, instruction flow proceeds to block 1020.
Block 1020 records a false alarm and instruction flow proceeds to block 1021. It should be appreciated that a false alarm is recorded, rather than a miss, because it suggests that the subject detected a change in the phoneme sequence when a change has not yet occurred. If, at decision block 1018, the hit window has begun, flow proceeds to decision block 1022.
At decision block 1022 a determination is made as to whether the hit window has passed. If the hit window has passed, prior to the subject releasing the animal 902, then flow proceeds to block 1026 where a miss is recorded, as described above. However, if the hit window has not passed flow proceeds to block 1024.
At block 1024, a hit is recorded for the subject. That is, the subject has correctly heard the target phoneme, and has released the animal 902 in an appropriate time frame. Flow then proceeds to decision block 1028.
At decision block 1028, a determination is made as to whether the subject has heard the target phoneme, and released the animal 902 within the hit window, three times in a row. If not, then flow proceeds back to block 1006 where another animal 902 is presented. If the subject has responded correctly, three times in a row, flow proceeds to block 1030.
At block 1030, the skill level for the selected tone sequence is increased by one level. In one embodiment, 18 skill levels are provided for each phoneme sequence. As mentioned above, the skill levels begin temporal modifications of the phonemes, and by separating the presented phonemes with an ISI of 500 ms. As the subject's ability to distinguish between the distractor and target phonemes improves, the temporal modifications of the phoneme is reduced to that of normal speech, and the ISI is reduced to 300 ms. One skilled in the art will appreciate that the degree of phoneme temporal manipulation, from 150% to 100%, the variation of ISI among the skill levels, and the number of skill levels provided, may vary depending on the LLI subject and the type of training that is required. In one embodiment, after a subject successfully passes a phoneme sequence with 150% time modification, and an ISI of 500 ms, the next skill level presented holds the time modification at 150%, but reduces the ISI to 400 ms. Flow then proceeds to decision block 1032.
At decision block 1032 a determination is made as to whether the maximum level has been reached for the selected phoneme sequence. That is, has the subject progressed through all the skill levels to the point that they are correctly recognizing a target phoneme with a duration of 100%, and with an ISI of 0 ms? If not, then flow proceeds to block 1006 where the animal 902 is again presented to the subject, this time, at an increased skill level. However, if the subject has reached the maximum level for a particular phoneme sequence, flow proceeds to block 1004 where a phoneme tone sequence is selected. If a subject has not yet played the new phoneme sequence that is selected, the skill level is set to the easiest level. However, if the subject has previously heard the new phoneme sequence, the level of play begins, either at or below the last skill level obtained, typically 5 skill levels below what was last obtained.
Selection of phoneme sequences and skill levels are performed by the program to insure that a subject is exposed to each of the phoneme pairs, but spends the greater portion of his/her time with those pairs that are the most difficult to distinguish. In addition, the number of recorded hits/misses/false alarms and reaction times are recorded for each level, and for each phoneme pair, on a daily basis. The records are then uploaded to a remote server where they are either reviewed by a remote SLP, or are tabulated and provided to a local SLP. The SLP then has the option of controlling the selection of phoneme sequence selection, and/or skill level, according to the particular needs of the subject, or of allowing automatic selection to occur in a round robin manner.
While not shown, the program also keeps track of the number of correct responses within a sliding window. This is visually provided to a subject by advancing the tractor 910, from the right to the left, for each correct response. After 10 correct responses, creative animations are played, and bonus points are awarded, to reward the subject and to help sustain the subject's interest in the game. Of course, the type of animation presented, and the number of correct responses required to obtain an animation are variables that may be set by an SLP.
Now referring to
The screen 1100 contains a number score 1102 and a stop sign 1104. The number score 1102 provides visual feedback to a subject regarding their progress in the game, and the stop sign 1104 provides a selection mechanism for ending the game. Also shown is a cat 1106. The cat 1106 provides animations for a subject during training. A grid 1120 is shown, in a 55 degree perspective, upon which are placed 3D tokens, further described below. In the center of the grid 1120 is an ear/hand button 1108. When a subject places a hand selector 1110 on top of the ear/hand button 1108, and selects the icon (by pressing a mouse key), then a trial in the Block Commander game begins. This is shown in
In
Now referring to
At block 1304 the game selects the first playing level that is to be presented to a subject. To the right of block 1304 is a table 1330 that illustrates the 5 processing levels that are used in the Block Commander game. The levels are distinct from each other in terms of the amount of stretching (in the time domain) that is used on speech, and the amount of emphasis that is applied to selected frequency envelopes within the speech. Flow then proceeds to block 1306.
At block 1306, the game presents a program to a subject that trains the subject to play the game. The training portion consists of 3 rounds. The first round trains the subject to distinguish between object sizes, e.g., large and small. The second round trains the subject to distinguish between object shapes, e.g., square and circle. The third round trains the subject to distinguish between object colors, e.g., blue, red, yellow, green and white. More specifically, the prompts given to a subject during training are:
For a subject to pass any of the training rounds, and progress to the next training round, two correct hits are required for each command prompt, with no errors. If an error is made, the score is reset, and play for that round starts over. All of the prompts for the training rounds are at processing level 1, 150% duration and 20 dB emphasis. After a subject has completed the training program he/she will not see it again. Upon completion of the training program, flow proceeds to decision block 1308.
At decision block 1308 a determination is made as to whether the training has been completed. If not, then flow proceeds back to block 1306 where training continues. If training has been completed, flow proceeds to block 1310.
At block 1310, a warm up exercise is presented to a subject. The warm up exercise is presented each time a user plays the game, at the speech processing level that was last completed. The warm up round includes the following prompts:
The ordering of the prompts is random each time the warm up is played. After presentation of each of the prompts flow proceeds to decision block 1312.
At decision block 1312, a determination is made as to whether the warm up round has been completed. If not, then flow proceeds back to block 1310 where the warm up continues. Otherwise, flow proceeds to block 1314.
At block 1314, an appropriate processing level is selected for a subject. The first time a subject plays the Block Commander game, processing level 1 is selected. However, after the subject has progressed beyond processing level 1, the level selected will be the level that the subject last played. Flow then proceeds to block 1316.
At block 1316, the first round of the game is presented to a subject. As mentioned above, in one embodiment of the Block Commander game, six rounds are provided. The rounds are as follows:
Each of the prompts are presented to the user in random order, but successful completion of each of the prompts in a round is required before a round is considered complete. After a first prompt is provided to a subject, flow proceeds decision block 1318.
At decision block 1318, a determination is made as to whether there have been 90% correct responses in a sliding group of 5 items. If not, then flow proceeds back to block 1316 where another prompt in a round is provided. If there have been 90% correct responses, as will be illustrated by 5 progress tokens at the bottom of the screen, then flow proceeds to block 1320.
At block 1320, the subject is shown a reward animation. In one embodiment, the animation consists of characters morphing out of the blocks on the board. Flow then proceeds to decision block 1322.
At decision block 1322, a determination is made as to whether the round is complete. A round is complete when a subject successfully responds to all of the prompts in the round. If the round is not complete, flow proceeds back to block 1316 where another prompt is provided to the subject. If the round is complete, flow proceeds to decision block 1324.
At decision block 1324, a determination is made as to whether all six rounds within the game have been completed. If not, then flow proceeds to block 1326 where the round level is incremented. Flow then proceeds back to block 1316 where prompts for the new round are presented. If decision block 1324 determines that all rounds have been completed, flow proceeds back to block 1314 where an appropriate skill level is selected. In one embodiment, if a subject successfully completes all six rounds, at skill level 1 (150% duration, 20 dB emphasis), he/she will progress to skill level 2 (125% duration, 20 dB emphasis).
The Block Commander program begins by providing a subject with a number of simple commands, stretched in time, with particular emphasis given to phoneme components that are difficult for an LLI subject to understand. As the subject correctly responds to the simple commands, the commands increase in difficulty. Once the subject masters the more difficult commands, the amount of stretching, and the amount of emphasis is reduced, and the process is repeated. The rounds continue, over the course of days and weeks, until the subject is correctly responding to the difficult commands at skill level 5, which is normal speech.
One skilled in the art will appreciate that the commands cause the subject, not only to understand the phonemes that are presented, but also to apply logical reasoning to the more difficult commands, and to recall the constructs of the commands. The requirement that the subject recall the command constructs is directed at improving the subjects memory, as well as to improving their ability to process acoustic events. It is believed that the games repetitive nature, that trains the subject's neurological connections to process speech, is also helpful in improving the subject's memory, and his/her cognitive skills in understanding linguistic relationships.
Now referring to
The screen 1400 contains a number score 1402, a stop sign 1404, and a progress element 1406, all within a circus ring environment. In addition, the screen 1400 contains a hand selector 1408, and an ear/hand button 1410. As in the Block Commander game, a user begins a test by selecting the ear/hand button 1410 with the hand selector 1408.
Referring to
At block 1604, the program begins presenting a random sequence of frequency sweeps to a subject. All sweep sequences are of the form: up-up; up-down; down-up; or down-down. Thus, if the program presents the sweep sequence “up-up”, a subject is to click on the left element 1502 twice. If the program presents a sweep sequence “down-up”, the subject is to click on the right element 1504, then on the left element 1502. So, once the program provides a sweep sequence to the subject, the subject selects the elements corresponding to the frequency modulated (FM) tone sequence. If the subject is correct, he/she is awarded points, the progress element 1506 advances upwards, and the ear/hand button 1410 is presented, allowing the subject to begin another test. During training, all upward sweeps are presented starting at 1 kHz and all downward sweeps ending at 1 kHz, with upward/downward sweeps at 16 octaves per second. The duration of the sweeps are 80 ms, and the sweeps are separated by 1000 ms. Research has shown that most LLI subjects are capable of distinguishing between frequency sweeps of this duration, and having an ISI of 1000 ms. After each sweep sequence is presented, flow proceeds to decision block 1606.
At decision block 1606, a determination is made as to whether the subject has correctly responded to 80% of the trials over a sliding scale of the last ten trials. If not, then flow proceeds back to block 1604 where the sequences continue to be presented. If the subject has correctly responded 80% of the time, flow proceeds to block 1608.
At block 1608, random sequences are again presented, at 1 khz, having a duration of 80 ms and an ISI of 1000 ms. Flow then proceeds to decision block 1610.
At decision block 1610, a determination is made as to whether the subject has correctly responded to 90% of the trials over a sliding scale of the last ten trials. If not, then flow proceeds to decision block 1612. If the subject has correctly responded to 90% of the trials over a sliding scale of the last ten trials, flow proceeds to block 1614.
At decision block 1612, a determination is made as to whether a subject has correctly responded to less than 70% of the trials, over a sliding scale of the last 20 trials. If not, indicating that he/she is responding correctly between 70-90% of the time, then flow proceeds back to block 1608 where the sweep sequences continue to be presented. If a determination is made that the subject is correctly responding less than 70% of the time over the last 20 trials, then flow proceeds back to block 1604, where the training begins again.
At block 1614, a 3-up, 1-down rule begins. This rule allows a subject to advance in difficulty level every time 3 correct responses are provided, while reducing the level of difficulty any time an incorrect response is given. Research has shown that a 3-up, 1-down rule allows a subject to obtain a correct response rate of approximately 80% near threshold, which is desired to motivate and encourage the subject to continue. A reduced accuracy rate discourages a subject, a situation that is not desired especially if the subject is an LLI child. Once the 3-up, 1-down rule is started, flow proceeds to decision block 1616.
At decision block 1616, a determination is made as to whether a subject has responded correctly the last 3 tests. If so, then flow proceeds to block 1620. If not, then flow proceeds to decision block 1618.
At decision block 1618, a determination is made as to whether a subject has incorrectly responded to the last test. If not, then flow proceeds back to decision block 1616 where another test is provided. However, if the subject has incorrectly responded to the last test, the difficulty level is reduced one level, and flow proceeds back to decision block 1616 where another test is presented. During the training level, all tests are performed at 80 ms duration, with 1000 ms ISI, which is the easiest skill level. Therefore, if the subject incorrectly responds at that level, no change in difficulty is made.
At block 1620, the skill level is increased. During training, the sweep sequences are presented at 1 khz, with 80 ms duration, but the ISI is reduced between the sweeps each time the level is incremented. In one embodiment, the ISI levels start at 1000 ms, and proceed through 900 ms, 800 ms, 700 ms, 600 ms and 500 ms. Flow then proceeds to decision block 1624.
At decision block 1624, a determination is made as to whether the ISI is at 500 ms. If not, then flow proceeds back to decision block 1616 where sweep sequences continue to be presented. If the ISI is 500 ms, the training session ends and the subject is allowed to enter the real game, at block 1626.
Referring now to
At block 1704, an appropriate skill level is selected. The skill levels used by Circus Sequence are shown in table 1730. For each of three frequencies: 500 hz, 1 khz, and 2 khz, a number of skill levels are provided. The skill levels begin by presenting frequency sweeps having a duration of 80 ms, and an ISI between the sweeps of 500 ms. As a subject advances, the ISI is reduced, either to 0 ms, or in one embodiment, to 125 ms. It should be appreciated that the ISI increments used should be selected to slowly train a subject's ability to distinguish between similar phonemes, such as /ba/ and /da/, while not frustrating the subject by training beyond levels required to distinguish between such phonemes.
When a subject first plays Circus Sequence, after passing training, he/she is provided with frequency sweeps beginning at 1 khz, having 80 ms duration and an ISI of 500 ms. On subsequent days, the frequency that is selected is random, and can be either 500 hz, 1 khz or 2 khz. Once the appropriate skill level has been selected, flow proceeds to block 1706.
At block 1706, a tone sequence is presented, according to the selected skill level. Flow then proceeds to decision block 1708.
At decision block 1708, a determination is made as to whether the subject has correctly responded to the last 3 trials. If not, then flow proceeds to decision block 1710. If the subject has correctly responded to the last 3 trials, flow proceeds to block 1712.
At decision block 1710, a determination is made as to whether the subject has incorrectly responded to the last trial. If not, then flow proceeds back to block 1706 where another tone sequence is presented. If the subject incorrectly responded to the last trial, flow proceeds to block 1714.
At block 1714, the skill level is decremented. If the skill level has an ISI of 500 ms, no decrease is made. However, if the skill level has an ISI that is less than 500 ms, the difficulty is reduced 1 level. For example, if the subject incorrectly responds to a trial having an ISI of 180 ms, for example, the difficulty level will be reduced, so that the next tone sequence will have an ISI of 185 ms. Flow then proceeds back to block 1706 where another tone sequence is presented.
At block 1712, if the user has correctly responded to the last 3 trials, the skill level is incremented. For example, if a subject is at a skill level with a sweep duration of 80 ms and an ISI of 250 ms, the skill level will increase such that the ISI for the next tone sequence will be 200 ms. Flow then proceeds to decision block 1716.
At decision block 1716, a determination is made as to whether the ISI is at 150 ms. If not, then flow proceeds to decision block 1720. If the ISI is at 150 ms, flow proceeds to block 1718.
At block 1718, the next lower duration is enabled. This allows the program to simultaneously trial a subject with multiple sweep durations, once the subject is successfully responding at an ISI level of 150 ms. For example, if a subject is correctly responding to tone sequences of duration 80 ms, with an ISI of 150 ms, then testing continues at 80 ms. In addition, testing is begun with sweep sequences of duration 60 ms, at an ISI of 500 ms. Flow then proceeds to back to block 1706 where another tone sequence is presented. This allows the program to present tone sequences of different duration, and different ISI, while tracking progress for each duration/ISI combination.
At decision block 1720, a determination is made as to whether the subject has reached a training threshold. In one embodiment, a training threshold is reached when the subject has had eight skill level reversals within six skill levels of each other. If such a threshold is reached, flow proceeds to block 1721. Otherwise, flow proceeds to decision block 1722.
At block 1721, the program moves the subject to the next frequency category to be tested. It is believed that once a threshold has been met on a particular day, the subject should not continue being tested at the same frequency. Thus, the program allows a subject to progress, either to an ISI of 0 ms (or some other minimal ISI) or to a threshold at one frequency, and then begin testing at an alternative frequency. Flow then proceeds back to block 1706.
At decision block 1722, a determination is made as to whether the ISI for a particular tone duration is 0 ms. If not, then flow proceeds back to block 1706 where another sweep sequence is presented. However, if a subject has reached a skill level of 0 ms ISI for a particular duration, flow proceeds to block 1724.
At block 1724, the program deletes the duration associated with the 0 ms ISI from the trial. This is because testing at that level is no longer required by the subject due to their proficiency. However, as mentioned above, an alternative embodiment may select an ISI of greater than 0 ms as the point where the duration is deleted from the game. Flow then proceeds back to block 1706 where more tone sequences are presented.
While not shown, in one embodiment, a threshold level is provided that causes the game to begin testing a subject at an alternate frequency. For example, if the subject is testing at 500 hz, and a threshold is reached, the program will begin testing the subject at 2 khz. The threshold is reached when a subject has 8 skill level reversals within 6 levels of each other. When this occurs, the program ceases testing at the frequency for which the threshold was reached, and begins testing at an alternative frequency.
Also, when a subject begins each day of testing, a frequency different than that tested the previous day is begun. Moreover, a skill level that is 5 less than completed the previous day is chosen, presuming the subject completed at least 20 trials for that frequency.
As mentioned above, each correct response causes the progress element 1506 to advance upward. After ten correct responses, a reward animation is provided to entertain the subject. When the animation ends, the subject is prompted with the ear/hand button 1410 to begin another trial.
Now referring to
The set of pictures 1802 are arranged into a 2×2 grid. When a subject selects any of the pictures, a word or phoneme is played. On any grid, there are two pictures that play the same word. Thus, for a 2×2 grid, there are two words that will be presented. The test for the subject is to distinguish between similar words, to recall which picture is associated with which word, and to sequentially select two pictures that present the same word. Similar words are presented together, with the words processed according to the processing levels shown in table 1902 of
Initially, subjects are presented words at processing level 1, with a duration of 150%, and having 20 dB emphasis of selected frequency envelopes within the words. In addition, different skill levels, as shown in table 1904, are provided that increase the grid size for a particular trial, and set the maximum number of clicks, or selections, that a subject can attempt before losing the trial. Operation of the game is illustrated in
Referring now to
At block 2004, a 2×2 grid is presented. The words associated with the 2×2 grid are selected from one of the four Word Groups shown above. The selection of the Word Group is random, except that tracking of previously played Word Groups is done to insure that all Word Groups are equally represented, and that a subject is not provided the same Word Group as played on an immediately preceding day. The words within a Word Group are typically selected according to their acoustic similarity.
The subject is required to sequentially select two pictures that have the same word associated with them. When a subject sequentially selects two pictures associated with the same word, the pictures are removed from the gird being played. After a subject completes a 2×2 grid, whether correctly or incorrectly, flow proceeds to decision block 2006.
At decision block 2006, a determination is made as to whether the subject has successfully passed three 2×2 grids. Referring to table 1904 of
At block 2008, a new grid is presented for a particular Word Group, or stimulus set. Initially, a 3×3 grid is provided, at skill level 2. The maximum number of clicks allowed for a subject to pass a 3×3 grid is 20. Within a 3×3 grid, 1 of the pictures is a wildcard, since there are an odd number of pictures. Selection of the wildcard simply removes the picture from the grid, and does not count against the subject as a selection, or click. After a 3×3 grid is presented to a subject, flow proceeds to decision block 2010.
At decision block 2010, a determination is made as to whether the subject passed the level. That is, did the subject properly distinguish between word pairs, and sequentially select picture pairs associated with words in 20 or less clicks. If so, then flow proceeds to block 2012. If not, then flow proceeds to block 2014.
At block 2012, the skill level is incremented. For example, if a subject was at level 2, he/she will increment to level 3. Note: levels 2-3 present a 3×3 grid with a maximum number of clicks of 20, while levels 4-7 present a 4×4 grid with a maximum number of clicks of 60. Once the skill level is incremented, flow proceeds to block 2020.
At block 2020, a grid according to the new skill level is presented. The grid is associated with the same Word Group that was previously used, but possibly with different words from the group. Flow then proceeds to decision block 2022.
At decision block 2022, a determination is made as to whether the subject has passed the level. That is, did the subject correctly associate the word pairs in less than or equal to the number of allowed clicks. If not, flow proceeds to block 2014. If the subject passed the level, flow proceeds to decision block 2024.
At decision block 2024, a determination is made as to whether the subject has reached skill level 7. Level 7 is termed the “decision” level. If the skill level that has just been passed is not level 7, then flow proceeds back to block 2012 where the skill level is incremented. However, if the skill level passed is level 7, flow proceeds to decision block 2026.
At decision block 2026, a determination is made as to whether all four stimulus sets, or Word Groups have been passed. If not, then flow proceeds to block 2018. However, if a subject has correctly passed skill level 7, for all four Word Groups, flow proceeds to block 2028.
At block 2028, the next processing level is selected. Referring to table 1902 of
At decision block 2030, a determination is made as to whether all processing levels have been completed. That is, has the subject reached processing level 5. If not, flow proceeds back to block 2004 where the game begins anew, with a 2×2 grid, but at the new processing level. However, if the subject has reached processing level 5, flow proceeds to block 2032.
At block 2032, a 5×5 grid is provided, with a maximum number of allowable clicks as 90. From this point forward, the game continues playing indefinitely, but the decision round, level 7, switches from a 4×4 grid to a 5×5 grid.
Referring back to decision block 2022, if a subject does not pass a particular level, flow proceeds to block 2014.
At block 2014, the skill level is decremented. Flow then proceeds to decision block 2016.
At decision block 2016, a determination is made as to whether the new skill level is less than level 1. Level 1 is considered a “slip” level indicating that if a user failed at this level, a new Word Group should be provided. If the skill level is not less than 1, flow proceeds back to block 2020 where a new grid is presented, according to the present level. If the new level is less than 1, that is, if the subject failed to pass a grid, at skill level 1, flow proceeds to block 2018.
At block 2018, the program discontinues presenting words from the present Word Group, and changes the Word Group used for the grids. Flow then proceeds back to block 2008 where a 3×3 grid is presented, at skill level 2, using words from the new Word Group.
The flow chart 2000 demonstrates that a subject is required to proceed from level 2 through level 7 for each of the four Word Groups, at a particular processing level, before he/she is allowed to advance to the next processing level. The progress creature descends with each click. If the creature reaches the bottom, then the grid is not passed. If all picture pairs are matched prior to the creature reaching the bottom, extra points are awarded, a reward animation is presented and the grid is considered passed. When a subject has correctly selected a predetermined number of picture pairs, the progress animal 1804 reaches the top, and the subject is rewarded by an animation.
Referring now to
As before, the screen 2100 contains an ear/hand button 2102 for beginning a trial, a stop sign 2104 for ending the game, and a number score 2106. Within the number score 2106 are five acorns, indicating the processing level currently being tested. Also shown are progress creatures 2114 indicating a number of correct responses. As a subject correctly responds to the game, a new progress creature 2114 is added. When the number of progress creatures 2114 reaches ten, a reward animation is provided and bonus points are awarded.
Referring to
Referring now to
At training block 2304 the subject is prompted to “press the ear button”. The prompting is processed at level 1 (duration 150%, emphasis 20 dB). Flow then proceeds to decision block 2306.
At decision block 2306, a determination is made as to whether the ear/hand button 2102 has been pressed. If not, then flow proceeds back to block 2304 where the prompting is repeated. If the ear/hand button 2102 has been pressed, flow proceeds to block 2308.
At block 2308, praise is played for the subject. Flow then proceeds to block 2310.
At block 2310, a single image appears in one of the two frames 2108, 2110, and a sound file pertaining to the image is played for the subject. Flow then proceeds to decision block 2312.
At decision block 2312, a determination is made as to whether the subject has selected the appropriate image. The image continues to be displayed until the subject selects the image. Flow then proceeds to decision block 2314.
At decision block 2314, a determination is made as to whether the subject has correctly selected the single image, three times. If not, then flow proceeds back to block 2310 where another image is presented, with its associated word. If the subject correctly selects an image/word combination three times, flow proceeds to block 2316.
At block 2316, a pair of images are presented, along with a command prompt containing a word associated with one of the images. The other image presented is termed the distractor image. The user must click on the correct image 4 out of 5 times in a sliding scale to start the game. After the double image is presented, flow proceeds to decision block 2318.
At decision block 2318, a determination is made as to whether the subject has correctly selected an image, from the image pair, in 4 out of 5 cases, on a sliding scale. If not, then flow proceeds back to block 2316 where another image pair is presented. Otherwise, flow proceeds to block 2320 where the subject enters the game. Flow then proceeds to block 2322.
At block 2322, a subject is presented a sequence of image pairs, with associated words selected from a particular processing set. The processing sets are chosen by grouping words having similar phoneme characteristics. Once all of the words have been presented within a processing set, flow proceeds to decision block 2324.
At decision block 2324, a determination is made as to whether the subject has correctly understood a word, and properly selected its associated picture from the picture pair with 90% or greater accuracy. If not, flow proceeds back to block 2322 where random selection of image/word pairs continue, until a 90% success rate is achieved. Flow then proceeds to block 2326.
At block 2326, a new processing set is selected. Flow then proceeds to decision block 2328.
At decision block 2328, a determination is made as to whether all of the processing sets have been completed. If not, then flow proceeds back to block 2322 where random selection of image/word pairs are presented from the current processing set. However, if all of the processing sets have been completed, flow proceeds to block 2330.
At block 2330, the processing level is incremented. Initially, the processing level is level 1. After a subject has completed all of the processing sets, with a 90% or greater accuracy for each of the sets, the processing level is increased to level 2. As described above, the duration of the words is decreased first, from 150%, to 125% to 100%, and then the emphasis of selected frequency envelopes is reduced, from 20 dB, to 10 dB, to 0 dB, until normal speech (level 5) is obtained. After the processing level is incremented, flow proceeds to decision block 2332.
At decision block 2332, a determination is made as to whether a subject has completed all of the sets at processing level 5. If not, then flow proceeds back to block 2322 where random selection of image/word pairs within a set are presented at the new processing level. However, if the subject has completed all of the processing sets at level 5, flow proceeds to block 2334.
At block 2334, Phonic Words continues to drill the subject randomly selecting image/word pairs within a processing set, at level 5.
Now referring to
For each phoneme pair, 26 different skill levels are provided, each level differing from the other in the degree of processing applied (duration and emphasis), and in the separation (ISI) of the distractor and target phoneme. Skill level 1 processes the phoneme pair by stretching the consonant portion 150% while leaving the vowel portion untouched, emphasizing selected frequency envelopes in the consonant portion 20 dB, and separating the distractor and target phonemes by 500 ms, for example. Skill level 26 provides a phoneme pair without stretching or emphasis, and with an ISI of 0 ms. Skill levels 2-25 progress towards normal speech by applying less and less consonant processing, with less and less separation between the distractor and target phonemes.
The screen 2400 contains an ear/hand button 2402 to allow a subject to begin a trial, a number score 2404 for tracking correct responses, a stop sign 2406 for exiting the game, a hand selector 2408, and progress elements 2410 for graphically illustrating progress to a subject. When the game is initially selected, five different animals are shown on the screen, each pertaining to a phoneme pair to be tested. A subject may select any one of the five animals to begin the game. After a subject has played the game with one of the five animals, the choice is reduced to four animals, and so on.
Referring to
Referring to
At block 2604, the game presents the screen shot 2400, and prompts a subject to “press the ear button”. Flow then proceeds to decision block 2606.
At decision block 2606, a determination is made as to whether the subject has pressed the ear/hand button 2402. If not, then flow proceeds back to block 2604 where the prompt is repeated, after a predetermined interval. If the subject has pressed the ear/hand button 2402, flow proceeds to block 2608.
At block 2608, the ear/hand button 2402 is presented, but this time without an audio prompt. Flow then proceeds to decision block 2610.
At decision block 2610, a determination is made as to whether the subject has pressed the ear/hand button 2402. If not, then flow proceeds back to block 2608. The subject remains in this loop until the ear/hand button 2402 is pressed. Once the ear/hand button 2402 is pressed, flow proceeds to block 2612.
At block 2612, a target phoneme, pertaining to a selected animal pair, is played for a subject. The target phoneme is processed at level 1, 150% duration, with 20 dB emphasis, as shown by the table 2640. Flow then proceeds to block 2614.
At block 2614, a single animal is presented that speaks the target phoneme. Flow then proceeds to decision block 2616.
At decision block 2616, a determination is made as to whether the animal that spoke the target phoneme has been selected. If not, flow proceeds back to block 2614 where the animal again speaks the target phoneme, after a predetermined interval. However, if the subject has selected the animal, flow proceeds to decision block 2618.
At decision block 2618, a determination is made as to whether the subject has correctly pressed the animal in ten trials. If not, then flow proceeds back to block 2612 where another trial is begun. However, once the subject has correctly responded in ten trials, flow proceeds to block 2620.
At block 2620, a target phoneme is again presented, at level 1 processing. Flow then proceeds to block 2622.
At block 2622, two animals are now presented, one speaking the target phoneme, the other speaking the distractor phoneme. The order of speaking the target and distractor phonemes is random, with the animal on the left speaking first, and the animal on the right speaking last. However, in this training level, the animal that speaks the target phoneme is visually highlighted for the subject. Both the target and distractor phonemes are processed at level 1, and are separated in time by 500 ms. Flow then proceeds to decision block 2624.
At decision block 2624, a determination is made as to whether the subject has correctly selected the animal speaking the target phoneme in 8 out of 10 trials, on a sliding scale. If not, then flow proceeds back to block 2620 where another trial is begun. If the subject has correctly responded in 8 out of 10 trials, flow proceeds to block 2626.
At block 2626, a target phoneme is presented to a subject, processed at level 1. Flow then proceeds to block 2628.
At block 2628, two animals are shown presenting a target phoneme and a distractor phoneme, both processed at level 1, with an ISI of 500 ms. The order of target/distractor phonemes is random. For this trial, however, the animal speaking the target phoneme is not visually highlighted for the subject. Flow then proceeds to decision block 2630.
At decision block 2630, a determination is made as to whether the subject has correctly responded to 8 out of 10 trials, on a sliding scale. If so, then the subject has successfully completed the training and flow proceeds to block 2634, allowing the subject to advance to the game. However, if the subject has not been successful in 8 out of 10 trials, then flow proceeds to decision block 2632.
At decision block 2632, a determination is made as to whether the subject has responded correctly less than 70% of the time in at least 10 trials. If not, then flow proceeds back to block 2626 where another trial is presented. If the subject has less than a 70% success rate, over at least 10 trials, then flow proceeds back to block 2614 where trials begin again, but where visual highlighting of the animal speaking the target phoneme is provided for the subject.
Referring now to
At decision block 2704, a determination is made as to whether the ear/hand button 2402 has been pressed. If not, then flow proceeds back to decision block 2704 until the subject chooses to hear the target phoneme. If the ear/hand button 2402 has been pressed, flow proceeds to block 2706.
At block 2706 a target phoneme is presented at an appropriate processing level. If this is the first time a subject has played the game, then the processing level for the phonemes is level 1, and the ISI between the target and distractor phonemes is 500 ms. Otherwise, the skill level pertains to the historical success of the subject, with the particular phoneme pair, as will be further described below. Flow then proceeds to block 2708.
At block 2708, two animals are shown, corresponding to the phoneme pair being tested, speaking the processed target and distractor phonemes, in random order. Flow then proceeds to decision block 2710.
At decision block 2710, a determination is made as to whether the subject has correctly selected the animal speaking the target phoneme. If not, then flow proceeds to block 2720. If the subject has correctly responded to the trial, flow proceeds to decision block 2712.
At block 2720, the skill level for play is decremented. For example, if the processing level is at level 1, having consonant duration of 150%, and emphasis of 20 db, but the ISI between the target and distractor phonemes is at 100 ms, the game will drop back to a skill level where the ISI is at 110 ms. However, if the skill level of play is already at level 1, then no change in processing is made.
At decision block 2712, a determination is made as to whether the subject has correctly responded in the last 3 consecutive trials. If not, then flow proceeds back to decision block 2704, awaiting another trial to begin. However, if the subject has correctly responded to the last 3 trials, flow proceeds to block 2714. It should be appreciated that the procedure illustrated in blocks 2710-2712 is the 3-up, 1-down rule, previously described in the Circus Sequence game above.
At block 2714, the skill level of the game is incremented. For example, if a subject has correctly responded to 3 consecutive trials, and is at a processing level of 100% duration, 20 dB emphasis, and an ISI of 0 ms, the next level of play will be at 100% duration, 10 dB emphasis, and an ISI of 500 ms. Flow then proceeds to decision block 2716.
At decision block 2716, a determination is made as to whether the highest skill level has been reached. If the subject has correctly responded to the last 3 trials, with no processing of the phonemes, and with minimal ISI between the target and distractor, then flow proceeds to block 2718. Otherwise flow proceeds to decision block 2722.
At decision block 2722, a determination is made as to whether the subject has reached a threshold. In one embodiment, a threshold is reached if the subject has had 8 skill level reversals within 6 skill levels of each other. If the subject has not reached a threshold, flow proceeds back to block 2704 where another trial is begun. If the subject has reached a threshold, flow proceeds to block 2718.
At block 2718, a new stimulus category is selected. That is, a new phoneme pair is selected for testing. Thus, if the subject has been tested with the phoneme pair ba-da, and has either mastered the pair by reaching the highest skill level, or has reached a threshold, then an alternate phoneme pair is selected, say aba-ada. Flow then proceeds back to block 2704 where a trial awaits using the new phoneme pair. In one embodiment, the skill level used for the new phoneme pair is selected to be 5 less than previously achieved for that pair. Or, if the subject has not yet been tested on the new phoneme pair, the skill level is set to 1. Testing continues indefinitely, or for the time allotted for Phoneme Identification on the subject's daily training schedule.
Referring now to
The stimulus that is provided to the subject is in the form of command sentences. The sentences are divided into 7 comprehension levels, with each level having between 4 to 10 groups of sentences. Each group has 5 sentences. For each stimulus sentence, a corresponding image is provided, with 1-3 distractor images. The subject is to listen to the stimulus sentence and select the corresponding image.
Each of the stimulus sentences may be processed by stretching words, or selected phonemes, in time, and by emphasizing particular frequency envelopes, as shown by table 3040 in
Referring now to
At block 2904, the subject is prompted to “press the yellow button”. That is, the ear/hand button 2802. Flow then proceeds to decision block 2906.
At decision block 2906, a determination is made as to whether the subject has selected the ear/hand button 2802. If not, flow proceeds back to block 2904 where the subject is again prompted, after a predetermined interval. If the subject has pressed the button, flow proceeds to block 2908.
At block 2908, the ear/hand button 2802 is presented, without audio prompting. Flow then proceeds to decision block 2910.
At decision block 2910, a determination is made as to whether the subject has pressed the button 2802. If not, then the subject stays in this loop until the button 2802 is pressed. Once pressed, flow proceeds to block 2912.
At block 2912, a subject is presented with a single image and corresponding audio stimulus. In one embodiment, the stimulus is processed at level 1, with 150% duration and 20 dB selective emphasis. Flow then proceeds to decision block 2914.
At decision block 2914, a determination is made as to whether the subject has selected the image corresponding to the presented stimulus. If not, then flow proceeds back to block 2912 where the subject is again prompted with the stimulus, after a predetermined interval. However, if the subject selected the image, flow proceeds to decision block 2916.
At decision block 2916, a determination is made as to whether the subject has correctly selected an image, 3 times. If not, then flow proceeds back to block 2912 where another image/stimulus combination is presented. However, if the subject has correctly selected an image, 3 times, flow proceeds to block 2918.
At block 2918, an image/stimulus combination is presented, along with a distractor image. Flow then proceeds to decision block 2920.
At decision block 2920, a determination is made as to whether the subject selected the appropriate image. If not, then flow proceeds back to block 2918. However, if the subject selected the correct image, flow proceeds to decision block 2922.
At decision block 2922, a determination is made as to whether the subject has correctly responded to 4 out of 5 trials, on a sliding scale. If not, then flow proceeds back to block 2918. If the subject has correctly responded 4 out of the last 5 trials, flow proceeds to block 2924 allowing the subject to start the game.
Now referring to
At block 3004 an image and stimulus combination is presented to the subject. In one embodiment, the game begins by selecting a group from Level 2, and then by randomly selecting one of the trials from the selected group. The processing of the sentence is performed at 150% duration with 20 dB selected emphasis. Flow then proceeds to decision block 3006.
At decision block 3006, a determination is made as to whether the subject correctly selected the image associated with the stimulus sentence. If not, the subject is shown the correct response, and flow proceeds back to block 3004 where another stimulus/image combination from the same group is presented. If the subject selects the correct image, flow proceeds to decision block 3008.
At decision block 3008, a determination is made as to whether all sentences within a stimulus set have been successfully completed. As mentioned above, the program begins in Level 2, by selecting a particular stimulus set for presentation. The program stays within the selected stimulus set until all stimulus sentences have been responded to correctly. The program then selects another stimulus set from within Level 2. If the subject has not correctly completed all sentences within a stimulus set, flow proceeds back to block 3004 where another sentence is presented. If the subject has completed all stimulus within a set, flow proceeds to decision block 3010.
At decision block 3010, a determination is made as to whether all sets within a particular comprehension level have been completed. If not, then a new set is selected, and flow proceeds back to block 3004. However, if all sets within a comprehension level have been completed, flow proceeds to block 3012.
At block 3012, the comprehension level is incremented. In one embodiment, a subject proceeds through comprehension levels 2-6, in order, with levels 7 and 8 interspersed within levels 3-6. Flow then proceeds to decision block 3014.
At decision block 3014, a determination is made as to whether all comprehension levels have been completed. If not, then flow proceeds back to block 3004 where the subject is presented with an image/stimulus combination from a stimulus set within the new comprehension level. However, if the subject has progressed through all stimulus sets for all comprehension levels, flow proceeds to block 3016.
At block 3016, the processing level applied to the stimulus sets is increased. The processing levels are shown in table 3040. For example, if a subject has just completed processing level 2, having a duration of 125%, and 20 dB emphasis, the processing level is incremented to level 3. This will present all stimulus at 100% duration, and 20 dB emphasis. In addition, it will reset the comprehension level to level 2, and will restart the stimulus set selection. Flow then proceeds to decision block 3018.
At decision block 3018, a determination is made as to whether all processing levels have been completed. If not, then flow proceeds back to block 3004 where a stimulus set from level 2 is presented to the subject, at the new processing level. However, if all the processing levels have been completed, the subject remains at processing level 5 (normal speech). Flow then proceeds to block 3020.
At block 3020, the comprehension levels are reset, so that the subject is presented again with stimulus from level 2. However, no alteration in the stimulus is performed. The subject will remain at processing level 5.
Study has shown that several weeks are required for a subject to advance through all of the comprehension levels, and all of the processing levels. Therefore, when a subject begins each day, he/she is started within the comprehension level, and stimulus set that was last played. And, the stimulus set will be presented at the processing level last played.
In Language Comprehension Builder, as in all of the other games, detailed records are kept regarding each trial, indicating the number of correct responses and incorrect responses, for each processing level, skill level and stimulus set. These records are uploaded to a central server at the end of each day, so that a subject's results may be tabulated and analyzed by an SLP, either working directly with a subject, or remotely. Based on analysis by the SLP, modification to training parameters within Fast ForWord may be made, and downloaded to the subject. This allows a subject to begin each day with a sensory training program that is individually tailored to his/her skill level.
The above discussion provides a detailed understanding of the operation of the present invention as embodied in the game modules within the program entitled Fast ForWord. Each of the game modules present different problems to a subject, using modified phonemes, frequency sweeps or speech commands that are stretched, emphasized or separated in time, according to the subject's ability, and according to predefined processing parameters within the program. Although alternative acoustic processing methodologies may be used, discussion will now be directed at algorithms developed specifically for use by the above described games.
In one embodiment, a two-stage speech modification procedure was used. The first stage involved time-scale modification of speech signals without altering its spectral content. The time scale modification is called the “phase vocoder”, and will be further described below. The second speech modification stage that was developed uses an algorithm that differentially amplifies and disambiguates faster phonetic elements in speech. “Fast elements” in speech are defined as those that occur in the 3-30 Hz range within an envelope of narrow-band speech channels of a rate changed speech signal. An emphasis algorithm for these fast elements was implemented using two methods: a filter-bank summation method and an overlap-add method based on a short-time Fourier transform. Both of these emphasis algorithms will be further described below.
Time-scale modification
Referring to
At block 3104, segmented digital speech input is provided to a processor. The segmented speech is assumed to be broadband and composed of a set of narrow-band signals obtained by passing the speech segment through a filter-bank of band-pass filters. The speech signals may be written as follows:
This is the convolution integral of the signal f(t) and h(t), a prototypical low-pass filter modulated by cos[ωn(t)] where ωn is the center frequency of the filters in the filter-bank, an operation commonly referred to as heterodyning. Flow then proceeds to block 3106.
At block 3106, the above integral is windowed, and a short-term Fourier transform of the input signal is evaluated at the radian frequency ωn using an FFT algorithm. The complex value of this transform is denoted:
fn(t)=|F(ωn,t)|cos[ωnt+φn(ωn,t)]
where φn(ωn,t) is the phase modulation of the carrier cos[ωn(t)]. Flow then proceeds to block 3108.
At block 3108 the amplitude and phase of the STFT is computed. It is known that the phase function is not a well behaved function, however its derivative, the instantaneous frequency, is bounded and is band limited. Therefore, a practical approximation fn(t) is:
where φ* is the instantaneous frequency. Flow then proceeds to block 3110.
At block 3110 φ* can be computed from the unwrapped-phase of the short-term Fourier transform. A time-scaled signal can then be synthesized as follows by interpolating the short-term Fourier transform magnitude and the unwrapped phase to the new-time scale as shown below.
At block 3112, a short-term inverse FFT is computed to produce digital speech output. This output is then provided at block 3114.
Filter-Bank Emphasis Algorithm
Now referring to
At block 3204, it is assumed that the speech signal can be synthesized through a bank of band-pass filters, as described above. This time, however, no heterodyning of a prototypical low-pass filter is used. Instead, a set of up to 20 second-order Butterworth filters with center frequencies logarithmically spaced between 100 and the nyquist frequency are used. The output of each band-pass filter resulted in a narrow-band channel signal fn(t). Flow then proceeds to block 3206.
At block 3206, we computed the analytical signal as follows:
an(n)=fn(n)+iH(fn(n))
where H(n) is the Hilbert transform of a signal defined as:
The Hilbert transform was computed using the FFT algorithm. It is known that the absolute value of the analytical signal is the envelope of a narrow-band signal. Thus, an envelope en(n) is obtained by the following operation:
en(n)=|an(n)|
The envelope within each narrow-band channel is then band-pass filtered using a second order Butterworth filter with the cutt-offs set usually between 3-30 Hz (the time scale at which phonetic events occur in rate changed speech). The band pass filtered envelope is then rectified to form the new envelope as follows:
ennew(n)=S(en(n)* g(n))
where
S(x)=x for x>=0, otherwise S(x)=0
and g(n) is the impulse-response of the band-pass second order Butterworth filter. Flow then proceeds to block 3208.
At block 3208, the signal is modified within each band-pass channel to carry this new envelope, as shown below:
Flow then proceeds to block 3210.
At block 3210 the modified signal is obtained by summing the narrow-band filters with a differential gain for each channel as follows:
where wn is the gain for each channel. The envelope is modified only within a specified frequency range from 1-10 KHz which normally spans about 16 channels. Flow then proceeds to block 3212.
At block 3212 segmented digital speech output is provided.
Overlap-Add Emphasis Algorithm
Referring to
At block 3304, the short-time Fourier transform is computed over a sliding window given by the following equation:
where h(n) is a Hamming window and the overlap between sections was chosen to be less than a quarter the length of the analysis window. The envelope can then be obtained within narrow-band channels from the absolute value of the short-time Fourier transform. The number of narrow-band channels is equal to half the size of the length over which the FFT is computed.
The energy of the envelope within critical band channels is then averaged, as shown:
where Cn is the corner-frequency of the critical-band channel n. At present, critical-band frequencies for children with LLI are unknown, therefore the present invention approximates the bands using parameters proposed by Zwicker. See E. Zwicker and E. Terhardt, “Analytical expressions for critical-band rate and critical bandwidth as a function of frequency,” J. Acoust. Soc. Ame., vol. 68, pp. 1523-25, 1980. As critical band frequencies for children with LLI become available, they can be incorporated into the present invention.
The envelope within each critical-band channel is then band-pass-filtered with cut off's set usually between 3-30 Hz with type I linear phase FIR equiripple filters. The band-pass filtered envelope is then threshold rectified. In contrast to the filter-bank emphasis algorithm, the modified envelope is added to the original envelope to amplify the fast elements while not distorting the slower modulations. This is given by the following equation:
where,
T(x)=x+1 for x>=0, otherwise 0
Flow then proceeds to block 3308.
At block 3308, a modified signal is obtained by summing the short-time Fourier transform using a weighted overlap-add procedure as shown below:
where g(n) is the synthesis filter which was also chosen to be a Hamming window. Flow then proceeds to block 3310.
At block 3310, windowing and over-lap addition for the algorithm is performed. Flow then proceeds to block 3312 where segmented digital speech output is provided.
Discussion of Improvements to “Fast Forward”
Many improvements have been made to the above described program to adapt it to the purposes described above. Among the many improvements, the below discussion will focus on the increased frequency and randomness of the reward structure of the program. That is, the reward structure, and animated surprises have been modified in each of the programs to further stimulate neuromodularity activity. By neuromodulatory structures, we refer to various subcortical nuclei that broadly project across the forebrain (e.g., the cholinergic basal forebrain, the dopaminergic ventral tegmental area and substantia nigra, the serotonergic raphe nuclei, and the noradrenergic locus coeruleus). The function of these nuclei is generally thought to be to modulate synaptic transmission and/or the overall activational state of the brain in response to behavioral needs (e.g., attentional state, alertness, success or failure in a task). The activity of these nuclei is under behavioral control in the normal state, and the renormalization of this function (i.e., bringing the activation and the effects of these nuclei back under proper behavioral control) is a key goal of our training exercises.
In general, in addition to the reward given for a correct response (as described above), there is at least a 30% chance that 2 rewards will be given for a correct response. In one embodiment, the double reward is given without regard to a pending animation. In an alternative embodiment, if a double reward coincides with a reward animation, a single reward is given. A summary of the reward improvements specific to each module is provided in the table below.
Referring now to
At block 3404, a determination is made as to whether the subject has correctly responded to a trial. If not, flow proceeds to block 3406 where no reward is provided. Otherwise, flow proceeds to block 3408.
At block 3408, randomization is performed to create a double reward. That is, in one implementation, a double reward is desired 30% of the time. This is provided within the computing system by utilizing a timer/counter that counts from 1-10. Upon a correct response to a trial, the timer/counter is examined. If the value of the timer/counter is in the range of 1-3, which it will be approximately 30% of the time, then a “yes” is provided to block 3410. If the timer/counter is in the range of 4-10, then a “no” is provided to block 3410. One skilled in the art will appreciate that there are many ways to generate a “random” value for the purpose of determining a percentage. The use of a timer/counter is merely one embodiment for obtaining a “random” or “percentage” other than 100% for double rewards. Once a value, or yes/no is determined by block 3408, flow proceeds to decision block 3410.
At decision block 3410, a determination is made as to whether a double reward should be presented. In one implementation, if the value is in the range of 1-3, the determination is yes, and if the value is in the range of 4-10, the determination is no. If no, flow proceeds to block 3412 where a single reward is presented. If yes, flow proceeds to decision block 3414.
At decision block 3414, a determination is made as to whether a reward animation coincides with the double reward. If a reward animation is to be presented, flow proceeds to block 3416 where a single reward is presented—even though the randomization indicated a double reward. If a reward animation does not coincide with the double reward, flow proceeds to block 3418. In an alternative embodiment, flow proceeds directly from decision block 3410 to block 3418 without the determination regarding the animation. That is, if the decision block 3410 determines that a double reward should be presented, then flow proceeds to block 3418 without regard to whether an animation coincides with the double reward.
At block 3418, a double reward is presented. Flow then proceeds to block 3420 where the double reward determination is done.
One skilled in the art should appreciate that the methodology illustrated with respect to
Although the present invention and its objects, features, and advantages have been described in detail, other embodiments are encompassed by the invention. For example, while no specific method has been described for modifying “punishments” associated with incorrect responses, surprise punishments, or unexpected animations may be implemented similar to the reward improvements discussed above. Furthermore, although the reward animations have been described with respect to a particular “random” methodology, one skilled in the art will appreciate that such an implementation is simply one way of disassociating the reward structure from user expectations. It is the surprise aspect of the reward structure, i.e., disassociating the rewards and/or the “value” of the rewards, from the correct responses, to which the present invention is directed. The actual frequency of the rewards, and the methodology to obtain the randomness is merely one way of achieving the disassociation.
Those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the spirit and scope of the invention as defined by the appended claims.
This application claims the benefit of the following U.S. Provisional Applications, each of which are hereby incorporated by reference in their entirety for all purposes: DocketSer. No.Filing DateTitleNRSC.010160/536129Jan. 13, 2004NEUROPLASTICITY TO REVITALIZE THEBRAINNRSC.010260/536112Jan. 13, 2004LANGUAGE MODULE EXERCISENRSC.010360/536093Jan. 13, 2004PARKINSON'S DISEASE, AGINGINFIRMITY, ALZHEIMER'S DISEASENRSC.010460/549390Mar. 2, 2004SENSORIMOTOR APPLIANCESNRSC.010560/558771Apr. 1, 2004SBIR'SNRSC.010660/565923Apr. 28, 2004ATP FINALNRSC.010860/575979Jun. 1, 2004HIFI V 0.5 SOURCE
| Number | Date | Country | |
|---|---|---|---|
| 60536129 | Jan 2004 | US | |
| 60536112 | Jan 2004 | US | |
| 60536093 | Jan 2004 | US | |
| 60549390 | Mar 2004 | US | |
| 60558771 | Apr 2004 | US | |
| 60565923 | Apr 2004 | US | |
| 60575979 | Jun 2004 | US |
