The following generally relates to imagery mnemonic and in particular to creating an imagery mnemonic.
Imagery mnemonic is a memory technique that employs a form of visual cue or prompt in order to help a user of the mnemonic remember a specific detail. The representation could be either directly or indirectly related to the idea that is trying to be memorized. The imagery mnemonic technique can be applied to tasks such as remembering lists, prospective memory and/or language learning. A difficulty of imagining an effective imagery mnemonic is dependent upon the creativity of the individual, and it can take a long time to visualize tasks in the form of composite images which trigger recall.
There is training available which helps to generate an imagery mnemonic; however, the training can be a lengthy process and is prone to failure, for example, due to the lack of creativity by the creator of the composed images and/or the lack of effort put into following the training by the user. A dynamically generated, memorable image can help a user to learn how to use the imagery mnemonic. However, the creation of memorable images is a difficult task, for example, at least because a definition of a memorable image varies from individual to individual.
Aspects described herein address the above-referenced problems and others.
In one aspect, a method for generating an imagery mnemonic includes receiving, via an input device of a computing system, at least two words of interest. The method further includes evaluating, with a processor of the computing system, the at least two words to determine what entities they represent, and identifying, with the processor, one of the at least two words as a subject word and another of the at least two words as an object word using a subject-object model. The method further includes searching, with the processor, a database of images for images corresponding to the at least two words. The method further includes identifying, with the processor, a classifier for the at least two words, and classifying, with the processor, the images including identifying a first image that includes the entity represented by the subject word and a second image that includes the entity represented by the object word. The method further includes creating, with the processor, an imagery mnemonic by combining the first and second images.
In another aspect, a computing system includes a memory device configured to store instructions, including an imagery mnemonic module, and a processor configured to execute the instructions. The instructions cause the processor to: receive at least two words of interest via an input device, evaluate the at least two words to determine what entities they represent, identify one of the at least two words as a subject word and another of the at least two words as an object word using a subject-object model, search a database of images for images corresponding to the at least two words, identify a classifier for each of the at least two words, classify the images from the search results, including identifying a subject image that includes the entity represented by the subject word and an object image that includes the entity represented by the object word, identify a location on the subject image for the object image, and create a composite image by merging the object image at the identified location on the subject image, wherein the composite image represents an imagery mnemonic.
In another aspect, a computer readable storage medium is encoded with computer readable instructions. The computer readable instructions, when executed by a processer, cause the processor to: receive at least two words of interest input via an input device, evaluate the at least two words to determine what entities they represent, identify one of the at least two words as a subject word and another of the at least two words as an object word using a subject-object model, search a database of images for images corresponding to the at least two words, identify a classifier for each of the at least two words, classify the images from the search results, including identifying a subject image that includes the entity represented by the subject word and an object image that includes the entity represented by the object word, identify a location on the subject image for the object image, and create a composite image by merging the object image at the identified location on the subject image, wherein the composite image represents an imagery mnemonic.
The invention may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating the embodiments and are not to be construed as limiting the invention.
The computing system 102 includes a hardware processor 104 (e.g., a central processing unit or CPU, a microprocessor, or the like). The computing system 102 further includes a computer readable storage medium (“memory”) 106 (which excludes transitory medium) such as physical memory and/or other non-transitory memory. The computing system 102 further includes an output device(s) 108 such as a display monitor, a speaker, etc., an input device(s) 110 such as a mouse, a keyboard, a microphone, etc. The illustrated computing system 102 is in communication with a local and/or remote image repository 112, which stores images.
The memory 106 stores data 114, such as images 116 and rules 118, and computer readable instructions 120. The processor 104 is configured to execute the computer readable instructions 120. The computer readable instructions 120 include an imagery mnemonic module 122. The imagery mnemonic module 122 includes instructions, which, when executed by the processor 104, cause the processor 104 to create an imagery mnemonic using images (e.g., the images 116, the image repository 112, and/or other images) based on words input via the input device 110 and the rules 118. The image repository 112 may be local and/or remote (e.g., a server, “cloud,” etc.) accessed over a network such as the Internet.
In one instance, the imagery mnemonic module 122 includes a “subject-object” model to define the input words and generate the imagery mnemonic for the words. This includes having the processor 104 search for a keyword, within the input words, that is known and likely to be a main focus of the mnemonic. The processor 104 labels this keyword as the “subject.” The processor 104 labels the remaining words as “object” words. Generally, a “subject” word is a trigger/cue word, which the user is most likely to remember, e.g., a task that the user performs regularly and/or other word likely to be in their long term memory, and an “object” word is a word the user is less likely to remember. An image of the “object” word is merged with an image of the “subject” word at a particular location, creating an imagery mnemonic. This includes identifying the particular location(s) on the subject image, which acts as a background/base image, and the object word(s) is merged to identified location(s). The “subject” word/image acts as a trigger to help the user remember the “object” word/image.
For this, the imagery mnemonic module 122 employs a trained classifier to classify images corresponding to “subject” and “object” words. An example of a suitable classifier is a cascade classifier, which is a statistical model built up in layers over a number of training stages. With each training stage, the model becomes more specific to a point where it only detects that which it has been training on and nothing else. In one non-limiting example, a Haar Cascade Classifier is trained using an Open Source Computer Vision (OpenCV) library. A Haar Cascade Classifier uses Haar-like features (e.g., rectangular, tilted, etc.) as digital image features for object recognition. Other classifiers are also contemplated herein.
For training, the classifier is first trained to learn what an entity (e.g., a “dog’) is with a set of images that include the entity. Then, the entity is segmented (e.g., into “eyes,” “paws,” “body,” “tail,” “nose,” etc.), and the classifier is trained to learn what the different segments are with the segmentations. Different classification trees are created for different entities (e.g., “dog,” “apple,” “tooth,” etc.), and the classification trees are stored locally and/or remotely in a searchable database or the like. When creating an imagery mnemonic, the processor 104 utilizes, locally and/or remotely, a particular classifier of the database associated with the input words, and merges the “object” image at the particular location of the “subject” image. The classifier is used to classify the “subject” image and to determine a region of interest (ROI) on the “subject” image where the “object” image is eventually merged. An outline of an “object” image could be used to facilitate merging images, e.g., without knowing which part is which and/or how the object was oriented.
The imagery mnemonics can be stored (e.g., in the memory 106, the image repository 112, etc.), conveyed to another device (e.g., via a cable and/or wirelessly over a network, portable memory, etc.), printed to paper and/or film, and/or otherwise utilized. For example, the imagery mnemonics can be incorporated with paper and/or electronic calendars, to-do lists, diaries, etc. For instance, a composite image based on tasks to occur can be attached to a calendar entry in a smartphone, an email application, etc. The imagery mnemonics can help a person visualize their own imagery mnemonic and/or be used as their imagery mnemonic, and/or can be used for training purposes.
At 202, the system 100 receives at least two words of interest, e.g., through speech and/or text via the input device 110. If the words are entered via speech, the entered words are recognized and converted to text through speech recognition software of the system 100 and/or imagery mnemonic module 122. The imagery mnemonic module 122 may include instructions for performing a spell check operation on the entered words to ensure at least two words are input.
At 204, the words are displayed via the output device 108 and accepted, rejected and/or changed via an input from the input device 110.
At 206, the processor 104 evaluates the at least two words to determine what entities they represent, including determining which word is a “subject” word and which word is an “object” word.
By way of a non-limiting example algorithm, the system 102 checks to see if there already is a classifier for either or both of the words. If there is a classifier for only one word, then the system 102 identifies the word with the classifier as the subject word (and hence identifies the subject image) and the other word(s) as the object word(s). If there is a classifier for both words, then the system 102 determines which word has been searched more by the user and uses that word as the subject word. If the words are equally searched, then the system 102 prompts the user to identify the subject word to the system 102.
In one instance, the processor 104 uses a lexical database of English nouns, verbs, adjectives and adverbs grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept, where the synsets are interlinked by conceptual-semantic and lexical relations, to determine a link between the at least two words. A non-limiting example of such a database is WordNet®. The order of the input can facilitate determining which word is a “subject” word and which word is an “object” word. Where the at least two words includes a sentence, the processor 104 can perform English parsing to interpret words such as “in” and “on.”
At 208, the processor 104 searches images in the images 116 and/or the image repository 112 for images that correspond to the entities represented by the at least two words. The search can be conducted using an application programing interface (API). A non-limiting example of an API is a Google Image Search application programing interface (API), which provides a JavaScript interface to embed Google Image Search results. Other API's include the Yahoo API, the Flickr API and/or other image search API. Alternatively, Google Custom Search, which enables creation of a search engine, can be used.
In one instance, image search API's are used as an image source for all of the images. For example, the images 116 are not searched. In this instance, the computing system 102 need not store the images 116. This may make the search process more flexible and imagery mnemonics can be made for any entities known. Where the images 116 are stored, the images 116 may include the user's gallery, which can be used to create the imagery mnemonics. This may help make imagery mnemonics more memorable and/or directly relevant to the context of the items being memorized.
At 210, classifiers are identified for the entities (i.e., the “subject” words). The classifiers are generated as described herein and/or otherwise.
At 212, the classifiers are used to classify the images from the search results, including identifying images including the entities represented by the words. The classification facilitates connecting or linking images to other images as it provides information about particular segments or sub-regions of an image.
At 214, the classified images for the entities are displayed via the output device 108.
At 216, a signal indicating an image is accepted or rejected or identifying a different image is received. Briefly turning to
Returning to
In one non-limiting instance, an overlaying strategy is used for the composition based on a region of interest identified on the “subject” image for the “object” image. The processor 104 can use the information learned about the words from WordNet®, etc. for the composition. The processor 104 can perform the composition using techniques such as Poisson blending, etc. to create images. Other techniques include magic wand, stamping, blending, layer masking, clone tool, chopping large images into components, warping, flip tool, opacity change, etc.
By way of non-limiting example, turning to
Turning to
It is to be understood that the selection of the “subject” and “object” words does not have to be at the start. In instances where the “subject” and “object” words are first selected, the “subject” and “object” words are used to know something about the characteristics of each component image to make the whole set easier to combine. Alternatively, later in the process, after selecting images relating to each word and running the classifiers, another word can be chosen as the subject word. Generally, analysis of the input words and/or the classifiers can be used to select “subject” words, which lead to images in which key features are easily identifiable so that other images can be connected to them.
It is to be appreciated that the ordering of the acts is not limiting. As such, other orderings are contemplated herein. In addition, one or more acts may be omitted and/or one or more additional acts may be included.
At 702, the system 100 receives input words “apple” and “tooth.” The two input words are processed in separate but similar processing chains 704 and 704′ as described next.
At 706 and 706′, the words are evaluated as described herein and/or otherwise to determine their meaning and to identify a “subject” image and an “object” image.
At 708 and 708′, images are retrieved for each of the two words as described herein and/or otherwise.
At 710 and 710′, the processor 104 checks to see if there is a classifier for each of the words.
If there is no classifier for one or both of the words, then at 712 and/or 712′ the images are displayed, and at 714 and/or 714′ the images are approved or rejected.
If approved for one or both of the words, then the accepted images are used for generating the imagery mnemonic at 716.
If rejected for one or both of the words, then acts 708 and/or 708′ are repeated for the rejected words.
If there is a classifier for one or both of the words, then at 718 and/or 718′, the images are classified.
If the classification fails for one or both of the words, then acts 708 and/or 708′ are repeated for the failed words.
If the classification succeeds for one or both of the words, then at 720 and/or 720′ the classified images are displayed, and at 722 and/or 722′ the images are approved or rejected.
If rejected for one or both of the words, then acts 708 and/or 708′ are repeated for the rejected words.
If approved for one or both of the words, then the accepted images are used for generating the imagery mnemonic at 716.
In another embodiment, the acts 720 and 722 and/or the acts 720′ and 722′ are omitted, and if the classification succeeds at 718 and/or 718′ for one or both of the words, then the classified images are used for generating the imagery mnemonic at 716, without user interaction and/or display of the images.
The method herein may be implemented by way of computer readable instructions, encoded or embedded on computer readable storage medium, which, when executed by a computer processor(s), cause the processor(s) to carry out the described acts. Additionally, or alternatively, at least one of the computer readable instructions is carried by a signal, carrier wave or other transitory medium.
The system and/or method describe herein is well-suited for applications such as, but not limited to, mental well-being to help a user visualize imagery mnemonics, a consumer calendar with auto generated images related to the content for each day would be a useful memory aid, home health care as part of a service, e.g., to help one remember day to day tasks, and education, e.g., for students who struggle to remember for their exams.
The invention has been described with reference to the preferred embodiments. Modifications and alterations may occur to others upon reading and understanding the preceding detailed description. It is intended that the invention be constructed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/EP2017/062463 | 5/23/2017 | WO | 00 |
Number | Date | Country | |
---|---|---|---|
62340809 | May 2016 | US |