Patent Application
20170169495
The invention relates to the field of accessing and retrieving electronic data. The system and method utilize an image acquisition device and a communication device to acquire and enter an image as a query in a database. Image recognition techniques then find related information in the database and return that information to the user.
The Internet began as a simple database of limited textual information, and quickly transformed into an extensive database of images, text, and audio information. It would take several lifetimes to hunt for various kinds of information throughout the Internet and USENET news groups, and, all the while, the number of files would be expanding faster than anyone's ability to peruse them.
Search engines were devised to manage the hunt. Search engines are programs that search the Internet for documents that contain specified keywords and return a list of documents which contain those keywords. These engines run programs called “spiders” that continuously explore the Internet and, often, USENET news groups, they index the information on websites that the spiders encounter. Indexing forms a vast database of website addresses that are associated with key words that have been found on the websites themselves.
Search engines such as Yahoo, Google, MSN, and International Business Machines' CLEVER require the user to enter at least one key term or query into a text field. Keywords, phrases, phrases in quotes, and Boolean queries are matched to various sites on the Internet, and when the query is complete a list of these sites is displayed for the user's review.
Although the most widely used search engines have a category that enables them to access images, none of them allows an image to be entered as a query or search entity. All known engines require that the user enter a text query, and the search hits files that display images that are associated with the entered text query. If a person sees an image and wishes to access online information about it, he or she will have to search for it using a text query. The user cannot use the image itself as a query. If the user cannot put his or her search request into words, he or she will not be able to conduct a search in a standard online search engine.
Several innovators are working to solve this need. Hewlett-Packard, for example, has developed a method of indexing an image that is based on information derived from a global positioning system (GPS). The system obtains an image along with its location, and indexes images according to their location. Such systems are useful in organizing album data since some digital cameras can acquire GPS data and correlate it with captured imagery. However, searching is limited to images that have a significant correlation with a given location.
A search engine developed by Xerox Corporation incorporates a multi-modal browsing and clustering system to retrieve image data. The system seeks similarities between images not only in textual references, but also in other associated information such as in-links, out-links, image characteristics, text genre, and the like. However, this engine is limited to specific image types which have defined colors, contain text, and have other visual identifiers. In short, the Xerox engine requires the images to have such specific characteristics, it limits the system's utility and viability as an all-purpose search engine.
Some attempts have been made to extract information from databases using images themselves as search entities rather than keywords related to the images. These systems can translate, provide information about, or interpret objects contained in an image. These systems generally work as follows. An input device extracts the object of interest from its background. The object is compared with objects stored in a pre-populated database to find a match. Finally, the system retrieves information in the database about the object and permits it to be displayed to the user. However, the system is limited to images containing extractable, defined objects, such as fruits, articles, animals, or any object which is easily outlined. However, many images require identification as a whole entity, such as an image of a geographic locations or a piece of artwork. As a result, this method has limited applicability.
Complex images with a myriad of superfluous objects are easier to identify using methods such as pixel analysis. Using this method, a database is populated with primitive, weighted vectors of images that facilitate the image processing. The inputted images are compared and matched through specific vectors that define them. Therefore, there remains a clear need for a system capable of capturing images, converting those images into computer readable formats, using the processed images as search queries in a search engine, comparing the images to images stored in the database, and, upon finding a match, displaying information associated with the image to a user of the system.
The present invention allows a user to extract information about an object, organism, or scenario of interest by acquiring its image and inputting that image into a search engine. The search engine can recognize the image and extract related information in the form of electronic data. Using this system, user can extract information about virtually anything, ranging from profiles of people of interest to historic information about a monument, or information about a piece of artwork. One object of the invention is the creation of a system which utilizes entry of an image as a search query or entity into a search engine.
Another object of the invention is the creation of a comprehensive registry of images, such as photographs, drawings, video clips, and holograms, which are associated with electronic data and serve as a universal image database that is available for matching images entered as search queries.
Another object of the invention is to provide the user of the system with the capacity to add information pertaining to an image to the database.
Another object of the invention is the creation of a system which utilizes pixel analysis as a means of comparing images entered as queries with images in the database in order to find a best-fit match.
A further object of the invention is the creation of a system which utilizes entry of an image along with alphanumeric characters to narrow the search. Boolean expressions (A AND, NOT, and OR B) can link images with text as a means of narrowing the search. Similarly, a plurality of images can be used in Boolean expressions. A further object of the invention is the creation of a system which utilizes entry of geographical coordinates in addition to the image in order to narrow the search. These coordinates can be entered by means of GPS, triangulation of cellular telephone towers, or the like.
Yet another object of the invention is the creation of an apparatus that employs image search and image matching to enable merchandising venues to reward shoppers for using their camera- and web-enabled cellphones and the like to post photos of their merchandise along with merchandiser-generated advertising and marketing information on their social networks.
Yet a further object of the invention is the creation of a system which utilizes entry of time and date of image capture along with the image in order to narrow the search.
An additional object of the invention is the creation of a system which utilizes entry of video clips or an image with a spoken word using Voice Recognition Technology (VRT) or a conventional keyboard to further narrow the search.
Another object of the invention is the creation of a system which utilizes the use of Optical Character Recognition (OCR) technology to read and interpret text associated with captured images, such that the text is entered as a search term accompanied by such images to narrow the search.
Another object of the invention is the creation of a system which enables advertisers or marketers to preplan response to the entry of images of advertisements by providing images of said advertisements or the products seen within to those who update the search engine, and links to relevant products, services, discounts, and the like.
Other objects of the invention are obtaining more information about products and services and, if desired, purchasing or leasing them. This object is enabled by the user's capturing of an image of a product or part of a product, the entry of said image as a search query, and the provision of links to commercial Web sites by those who update the search engine.
Another object of the invention is the creation of a system to aid education. In this embodiment, the user captures an image and obtains information about the subject of the image from online educational sources such as books, encyclopedias, dictionaries, translators, and the like.
Another object of the invention is the creation of a system which enables the user to communicate with at least one person. In this embodiment, the user captures an image of a person of interest and obtains contact and other information posted online by or about the person of interest. The person of interest may be observed “live,” in a photograph or video, projected onto a surface, or on an electronic display, such as a display of a page of an electronic social networking service.
A further object of the invention is the creation of a system which can act as a travel guide, which gives the user the capacity to capture an image and obtain information such as location, translation, historic description, current news, nearby attractions, where to stay, where to eat, transportation, current currency exchange, and the like.
In accordance with one embodiment the present invention comprises a system for accessing electronic data by providing an image comprising: (i) a means for capturing an image, (ii) a means for transmitting said image to a database wherein the database comprises: a. a means to receive said image, b. a means to access electronic data associated with said image, and c. a means to transmit said data to a display unit.
In accordance with another embodiment the present invention comprises a method of extracting electronic data from a database by providing an image captured by capturing means comprising: (i) providing computer coded images stored on the database and further linked to electronic data, (ii) entering captured image, (iii) performing image recognition functions to computer code said captured image, (iv) matching said computer coded image to said computer coded images stored on the database, (v) linking said captured image to said matched linked electronic data, and (vi) presenting said electronic data on a display unit.
A further understanding of the present invention can be obtained by reference to embodiments set forth in the illustrations of the accompanying drawings. Although the illustrated embodiments are merely exemplary of systems for carrying out the present invention, both the organization and method of operation of the invention, in general, together with further objectives and advantages thereof, may be more easily understood by reference to the drawings and the following description. The drawings are not intended to limit the scope of this invention, which is set forth with particularity in the claims as appended or as subsequently amended, but merely to clarify and exemplify the invention.
Detailed illustrative embodiments of the present invention are disclosed herein. However, techniques, systems and operating structures in accordance with the present invention may be embodied in a wide variety of forms and modes, some of which may be quite different from those in the disclosed embodiment. Consequently, the specific structural and functional details disclosed herein are merely representative, yet in that regard, they are deemed to afford the best embodiment for purposes of disclosure and to provide a basis for the claims herein that define the scope of the present invention. The following presents a detailed description of a preferred embodiment of the present invention.
The present invention provides a system capable of capturing images, entering the images into the search engine, extracting information associated with the images, and presenting the information to a user. Image capturing devices 100 capture the image and then transfer the image to communication devices 101 having transmitting and receiving means capable of communicating with database 103 through network 102, as shown in
Network 102 can be of any type, including but not limited to a network that is wired, wireless, GSM, ISDN, Ethernet, CAN, Wi-Fi, LAN, Bluetooth, or the like. Likewise, the capturing apparatus can be any device capable of transferring a real-time visual entity into a digitalized image such as, but not limited to, digital/analog cameras, video cameras, scanners, hand-held scanners, camera-enabled cellular telephones, camera-enabled PDA's, or the like. The communication device can be any device or combination of devices having communication functions and displaying means such as, but not limited to, a hand-held device, cellular telephone, hybrid cellular telephone/PDA device, PDA, remote server, RFID device, Internet accessible camera, personal computer, laptop computer, pocket computer, hybrid electronic device, or the like. The image-capturing device can be connected to a communication device through a hard-wired data link, wireless data link, or any other type of connection. Many image-capturing devices and communication devices are integrated into one unit, or can be integrated into one unit, such that any communication device can have image capturing capabilities and vice versa. The combination of the plurality of image-capturing devices and the plurality of communication devices will be referred to as CI devices (Communicable-Imaging Devices) hereinafter due to the difficulties with making definite distinctions between these devices.
A possible method of component interaction and the associated processes are depicted in
The disclosed system utilizes image recognition technology to define an image and retrieve information about it from a large database. Many techniques can be used for image recognition as it has been an emerging field since the mid-1900s. The most widely-used approach for image recognition is object extraction, as shown in
Then the image 202 can be segmented in process 231 into contiguous regions where the result is seen on the segmented image 203. The next step in the imaging process is to filter image 203, or perform low-level extraction in process 232, in order to completely define object 211 from the background 210. Once extracted and enhanced the object's lines 220 are located in image 204. Following, vectors are assigned to the extracted lines and the image is stored in process 233 as a series of vectors (matrices) that are compressed and quantized to a finite amount, which often causes loss of data and, consequently, resolution, when and if the image is later viewed. It is contemplated that the order of image processing steps (e.g. noise filtering and segmentation), the number each step is performed, and the addition of further processing steps can vary with each application without departing from the spirit of the present invention. After the object's lines 220 are defined, stored, and compressed, the mathematical representations are compared to other mathematical representations of images in a database. These mathematical representations might differ slightly due to this loss of additional information during processing. Therefore, when compared in a database, the information returned to the user will most likely need to contain a plurality of best-fit matches. This process of feature extraction and comparison is called the Digital Elevation Model (DEM) for image registration.
In addition to the improvements being made to the current processes of pattern recognition, image recognition, and other types of computer vision, new methods are being developed to troubleshoot problematic areas of the pre-existing ones. For instance, there are manners of extracting image data by texture, color, neural networks, location, background objects, and the like. However, these areas still require improvement for reliability. Nevertheless, the present invention envisions future applications for potential use of these new technologies as the image-recognition process in this invention.
The information associated with the images varies with different system applications. The source of information can comprise a single service provider site, combination or network of sites, or the entire universe of available information on the Internet. In a single site and a single application, each image or a group of images is linked to preset information. Essentially, each image or a group of images can have a webpage associated with it. For example, a user enters image 301 to search engine 300, shown in
If multiple sites of information are used, the images stored in the database can be indexed with text identifiers or the like, such as an image title, titles, or names of objects in the image. If the user enters image 301 to search engine 300 of
Consequently, the user could be presented with more source of information than he or she needs. To narrow the field of the search, the user can specify particular information she desires within the scope of system application through the use of Boolean expressions as illustrated in
Additionally, the search engine 300 could function such as disregarding the Boolean inputting field 400 and using a default Boolean parameter to search the database. In such case, it is preferred and common in practice to use the AND parameter. Also, the utilization of OCR technology to achieve a more automated system is possible. The system can transform images with embedded text into key words and enter those key words as search terms for the search engine, further shaping the extent and nature of the search. Alternatively, a series of alphanumeric characters, such as key words, is generated and entered by a user to further clarify and narrow the search. For example, a traveler can take a photo of the window of a restaurant, capturing parts of a menu, parts of the window display, or the name of the restaurant. When the photo is entered into a search engine, it returns information pertaining to reviews, decor, value, history, or the like.
When applying the system to multiple applications, the database search can be arranged into categories as shown in
If the CI device is wireless, a wireless transmitting device, such as a remote tower, is used to transfer the information from the CI device to a network, the process of its operation is shown in
The present invention has an important applicability in the commercial sphere. The ability to capture a product image or an image related to a product, acquire information about it, and purchase it by means of the CI device is desirable. The CI device might capture an image of a product in a store or of a product of interest in the possession of another party. Alternatively, the user can capture a product image from another image, such as a pamphlet, TV commercial, monitor of a computer, screen of a hand-held device, magazine, newspaper, product label, poster, or the like. Furthermore, the system enables a user to capture an image of any person, place, or thing, to receive information about the object, and to take a subsequent action such as making a purchase, leasing a product, arranging financing, or arranging delivery or pick-up of the product. When capturing an image of product labels, various printed indicators can be useful for fast and accurate image recognition. Barcodes, serial numbers, model numbers, or any other identifying parameter can help to identify the product, since they are unique. Examples of commercial applications include, but are not limited to, real estate, retail stores, entertainment, and other such venues.
The user of the system may be interested in the product, but not have the inclination to review the information about it as soon as it is retrieved due to time, money, and/or availability constraints. To accommodate for this, the system allows the user to capture an image of the product and store it for later use. Additionally, one might capture a desired product with an unwanted detail, such as color, size, or the like, and use the system to identify the product. Upon identification of the product, the user can then access additional information about availability of variations of the product and locations to purchase it.
The product information associated with product 901 might consist of, but is not limited to, product description 911, pricing, store locations and availability, online purchase capabilities, purchase statistics, information about related products, and the like. Additionally, the information might consist of links to a plurality of retail store sites 912, product manufacturers, online stores 913, online auction sites, and the like. After reviewing the product information, the user is able to purchase the product using the acquired information. Alternatively, after capturing an image and instead of acquiring product information, the CI device is directed to an order placement site wherein the user can readily place an order. Preferably, each user of the system has a personal profile such that the system can acquire information according to the user's criteria. The profile might consist of price limits, residency, taste, sizes, and the like. In another embodiment, providing the system with the residence or workplaces of a user allows the search engine to extract proximate store locations. Moreover, the user might enter a current location, or the system might have positioning capabilities such as GPS to find proximate locations to the user at the time of image capture. Also, information stored such as clothing sizes, either in the CI device or in a remote database, enables the system to extract only the locations having the correct items in stock. The personal profile might also include the user's asset information, facilitating payments and/or refunds. There has been recent speculation that cellular telephones will assume functionality of credit cards, identification means, access means, and the like. This functionality certainly is adaptable to the presently disclosed system.
Advertising is another commercial application of the present invention. For example, a user captures an image of an advertisement in a magazine, on a poster, or on the screen of a television, transmits the advertising image to a database, and acquires additional information about the product, commodity, or service. The user may also be linked to the source site of the advertisement. Advertisements might be captured from pamphlets, flyers, newspapers, books, posters, magazines, newspapers, TV commercials, coupons, or the like. Alternatively, information about services involving matters of health, law, travel, insurance, and the like also may be acquired. For example, a person can “shoot” a movie poster or marquee to obtain reviews of a movie, times and places of showing, cost of tickets, information about the director and actors, and information about other movies that might appeal to the user. The user of the system can also purchase tickets.
When capturing an image of an advertisement, various printed or on-screen indicators can be useful for fast and accurate image recognition. Barcodes, two-dimensional barcodes, two-dimensional figures, watermarks, digital watermarks or any other unique identifying parameter can help to identify the advertisement, since they are unique. When the user captures an image of the advertisement having the unique identifying parameter, the CI device sends the image to the database, wherein the image is processed and compared with images stored in the database. Relevant images stored in the database may comprise the entire advertisement having the unique identifying parameter, the advertisement without the unique identifying parameter, or simply the unique identifying parameter itself. At least one of these database images is associated with the information sought by the user. When the at least one image comprises the unique identifying parameter, the user can be taken directly to the source site or source site may be listed as a hit. The system can be set such that the database ceases processing additional parts of an image upon recognition of a unique identifying parameter and accesses the information associated with said unique identifying parameter. In case of recognition of two or more unique identifying, the information associated with each parameter can be listed as a hit. Advertisers and marketers can also induce users to capture an image of an advertisement by including a unique identifying parameter in the advertisement.
On the other hand, in the context of social networking, the person of interest may seek to find people with common interests; therefore, his or her profile 1110 may contain contact information that enables the inquiring user of the system to establish contact. The contact information can be a phone number, an address, an e-mail address, an instant messenger screen name, or an anonymous contact capability. For example, if a instant messenger screen name is available, the user uses CI device 1101 to send instant message 1111 to person of interest 1100 through the Internet, and person of interest 1100 decides whether or not to respond. For security purposes, the system may request that the user first transmit his or her profile to the person of interest, providing a basis for the person of interest to decide whether or not to maintain or expand contact. The user of the system may also use CI device 1101 to capture an image of multiple individuals in a facility. The system may provide the user with a selective choosing device, such as a scroll button, a mouse, or a numbering system, to select persons of interest and to acquire their information.
In another embodiment, an electronic social networking service will have a database that contains a plurality of images of people and information regarding the plurality of people. A user of the social networking service may seek to identify or communicate with an unidentified person of interest 1100 in a photograph or a group photograph on an electronic display. Such a display may comprise the screen of a desktop computer, a laptop computer, a tablet computer, a smartphone, a hybrid device, or the like. The user may capture an image of the person of interest by clicking on his or her image on the display, by touching the image, or by outlining the borders of the image with a mouse, stylus, or finger. Capturing the image of the person of interest may automatically trigger a search for the person matching the image on the website.
Alternatively, upon capturing the image, the user may institute the search by using keyboard commands or by clicking on or touching a feature of the website whose function is to commence a matching process in which the image of the person of interest is compared with the plurality of images in the social networking service database 1105. Such a feature may have a name such as “Find Me,” “ID Me,” “Who Am I?” Contact Me,” or the like. If the processing means of the database finds a match, a message is generated to the person of interest, indicating that a user is seeking information about him or her. The user is informed if no match is found. If a match is found, the person of interest may accept the invitation of the user, may decline, or may seek information about the user before authorizing the release of personal information. The user may then be informed that the person of interest accepts the invitation to provide information and initiate communication, declines the invitation, or seeks to view the user's profile before deciding whether or not to accept the invitation. The message sent to the person of interest will include information that allows him or her to respond anonymously to the user. The person of interest may also view the user's profile anonymously. One way in which the person of interest may anonymously view the user's profile is if a link to the user's profile is embedded in the user's initial message to the person of interest, and if the person of interest may access the user's profile without leaving a record of having done so that is available to the user.
Another object of the invention is to enable the person of interest to utilize the database of at least one online matching service, online dating service, social network or social medium (collectively referred to hereinafter as “online network”) to find users who request communication with him or her based on his or her image or images. In practice the person of interest may upload at least one image of himself or herself to the online network, or the person of interest might upload a plurality of images to enhance the probability of a match. The method of entry of images of the person of interest may be by means of using a smart phone application (or an “App”) with his or her mobile phone, phablet, tablet, laptop computer, desktop computer, or similar device. The person of interest would capture at least one image of himself or herself by means of the App or a smart phone camera App, and then uploading the image or images to the online network database. Alternatively, the person of interest may scan, copy, or take a photo of at least one image of himself or herself from a printed matter or an electronic display and upload the image or images to the database. The online network may provide the person of interest a window for entry of the image or images. Following entry of the image or images associated with the person of interest, the image or images are stored in a database that is linked to the online network. It is also possible that the image or images are uploaded to the database when the person of interest, or a third party, tags the image or images. The online network may use a method of tagging that does not identify the person of interest when the requesting user first requests communication, but, rather, enters the image into the database with a link that permits a requesting user to relay a message to the person of interest.
In practice, a requesting user uploads at least one image associated with a person of interest to an online network to establish communication with the person of interest. The online network may provide a window for entry of the image or images. The requesting user may be required to log on or sign into the online network to access the window that enables uploading of the image or images. Upon entry of an image by the requesting user or the person of interest, the online network may inquire as to whether either party desires to enter another image. The online network database may also be linked to at least one other database to broaden the access to the person of interest's image or images. Upon entry of his or her image to the online network database, the person of interest may be requested by the online network to create or enter a user ID and password, or a biometric ID, such as a fingerprint, to sign onto the online network in order to ascertain whether at least one requesting user has sent a message to him or her in response to viewing his or her image or images. Alternatively, the person of interest may already be logged onto the online network, in which case the person of interest may find a message about a requesting user when he or she loads the online network. A user ID or biometric ID can also be used by the person of interest to access his or her images stored in the network database for purposes of editing them or adding or deleting images.
A requesting user may upload an image or images of the person of interest from one or more of a plurality of means, such as by taking a photo of the person, or scanning, photographing, or capturing an image of the person from an electronic display or printed photo, or by tagging an untagged photo on an online network. It is thus envisioned that the online network might provide at least two windows to enable the service: one that is used by a requesting user and one that is used by a person of interest. On different occasions the requesting user can become a person of interest, and vice versa. The processing means of the network then compares the image or images entered by the requesting user to the image or images uploaded by the person of interest and stored in the database. In the event of a match or a “hit,” the online network confirms matching the photo to the requesting user and invites the requesting user to send a message (possibly including a brief user profile-perhaps standardized for a first contact) to the person of interest. The message may also be sent automatically upon the event of a hit. The message may include personal information about the requesting user, such as photos, videos, a physical description, demographic information, level of education, financial information, religious information, interests, a link to the user's online network home page, and the like. Alternatively, the online network may provide a standard data packet to be filled in or completed by the requesting user to accompany messages that express a desire to communicate with the person of interest.
If upon receiving the message from the requesting user, the person of interest does not desire to pursue further exchanges of information with the requesting user, the exchange process can stop here. Upon receiving a message from the person of interest that he or she does not wish to pursue an exchange of messages, the online network may send the user a standardized message to indicate that the person of interest chooses not to move forward. If the person of interest is interested in pursuing exchanges of information, he or she can return a data packet of his or her own or else request more information from the requesting user before transmitting his or her data packet. The exchanges can continue until the requesting user or the person of interest decides either to identify himself or herself, to provide a link to his or her page on the online network, to make an arrangement to talk or meet with the other party, or to end the exchange of information-that is, to discontinue the back and forth and the possibility of developing a relationship. At any point in the exchange process, the online network might provide a feature that can be checked or otherwise accessed to send a message that declines further exchanges.
Another preferred embodiment involves the use of image search and image matching by an apparatus (a system involving hardware and a computer or mobile phone application, or app) that enables merchandising venues to reward shoppers for using their camera- and web-enabled CI Devices (devices having cameras, processors, displays, transmitters, receivers, and location means, such as smartphones) to post photos of their merchandise and additional merchandiser-generated information (annotations, or tags) on social media platforms (also known as social networks or social media websites). Shoppers' posting of the merchandise along with the additional information advertises and markets the merchandise for the merchandiser. The additional information can comprise the identity of the merchandise, the identity of the merchandising venue, and the location of the merchandising venue. The merchandising venue may also include supplemental information selected from the group consisting of contact information regarding the merchandising venue, the opportunity to purchase or lease the article of commerce from the merchandising venue, the opportunity to purchase similar articles of commerce from the merchandising venue, information from related merchandising venues, discounts, rewards, an icon, humanly recognizable information, machine-readable information, and any combination thereof. Information beyond the identity of the merchandise, the identity of the merchandising venue, and the location of the merchandising venue may be too lengthy or complex to display along with the image; however, the availability of the additional information can be made known to viewers by means such as icons, hot spots, hashtags, symbols such as QR codes, and in-image methods such as moving a mouse over the image to reveal an overlay of a segment of the image or to produce a pop-up box, or to click on the image, thereby directing the user to a new page. If viewers open the image and at the least some additional information about the image on their desktop, laptop, or tablet computers, they can access the supplemental information associated with a QR code or other symbol through their web-enabled CI Devices and the like. The merchandising venue or the operators of the apparatus determine what information comprises the at least some additional information and what information will comprise extended supplemental information.
A merchandising venue can comprise a restaurant, a store, a retail store, a clothing store, a department store, a grocery store, a supermarket, an electronics store, an office supplies store, an auto dealership, a convenience store, a museum, a gallery, a bookstore, a wholesale distributor, a retail distributor, a theater, a stadium, an online store, a virtual store, and any combination thereof. Merchandising venues and shoppers both register with the apparatus. The merchandising venue uploads images of their merchandise to a database of the apparatus, annotates or tags the images to identify and locate the merchandise, and may further annotate or tag the images with supplemental information to further advertise and market the merchandise. Once shoppers have registered with the apparatus, the apparatus can use shoppers' web-enabled CI Devices' location means to identify the merchandising venue; shoppers may also permit the apparatus to use the web-enabled CI Devices' cameras to take and automatically post photos (images) with merchandiser-generated annotations or supplemental information on shoppers' social media platforms. Alternatively, the apparatus displays the image and the merchandiser's additional information regarding the image on the user's display, and the user selects the social media platform or platforms on which to post the image along with the merchandiser's annotations regarding the image. The apparatus uses image matching to match shoppers' photos with annotated or tagged images of merchandise that have been uploaded to the database of the apparatus. Shoppers select platforms on which they will post the annotated photos, following which the apparatus posts the photos and the associated tags on the shoppers' social media platform. Annotations or tags can include information such as the identity of the merchandise and the name and address of the merchandising venue. Another option or feature allows for the apparatus to retrieve other merchandizing venue information from platforms such as Yelp and Google Maps. An associated icon or link can be made to appear in order to enable the user to click and follow this icon or link to Yelp or Google if desired. Depending on the social media platform, the users' social media audience will have the possibility to follow a link or hashtag in order to direct themselves to an information hub to see additional information such as details about the merchandise, alternate views of the merchandise, possible sales, discounts, the arrival of new related or updated merchandise, merchandiser contact information, electronic ordering, and various other rewards and opportunities.
A humanly recognizable or machine readable feature such as an icon or a hashtag, even a QR code, may be included in the annotation or tag to make viewers of users' social media platforms aware of supplemental information such as details about the merchandise, alternate views of the merchandise, possible sales, discounts, the arrival of new related or updated merchandise, merchandiser contact information, electronic ordering, and various other rewards and opportunities. The photo and/or the icon can be touched to yield the supplemental information.
When shoppers register with the apparatus, they permit the apparatus to use their web-enabled CI Device cameras and location means upon opening the app. Shoppers can then open the apparatus and take photos with their web-enabled CI Device cameras; following image matching and automatic annotating or tagging, the images appear on the shoppers' screens along with the merchandiser's annotations or tags; shoppers may then select their preferred social media platforms and post the images along with their annotations or tags on the platforms. The apparatus then transmits a message to shoppers, indicating the merchandiser's reward. The posting of annotated or tagged images on shoppers' social media platforms advertises and markets the merchandise for the venue, manufacturer, or distributor, as well as publicizing the venue itself.
Alternatively, users can set the apparatus to automatically post their photos along with merchandiser-generated annotations to specified social media platforms. In this particular embodiment, users do not view the merchandiser's annotations or tags before they are posted along with the images. Users can simply open the apparatus (app), take the photo, and receive the reward. The photo is automatically tagged and posted on the specified social media platform. The photo with its tag can also be displayed simultaneously on the user's display (such as a smartphone screen). In order to allow users to view their images and the merchandiser-annotated annotations before they are posted on their social media platforms, the apparatus can also provide an intermediate step in which it displays the photo and its annotation on the user's display. The apparatus may request permission to post the image and tag on the user's social media platform. The users may have the opportunity to press a button or a hot spot, for example, to post the image and annotation. In this example, users can also be given the opportunity to change social media platforms, not to post at all (that is, to cancel), to save the image, to save the image along with the merchandiser's annotations, to add his or her own comment to the image and tag (e.g., “Best quail eggs ever!” “This app is pretty cool—I get a free dessert next time I come here,” “Celebrating Mark's birthday”), and any combination thereof. Users are not rewarded unless they post the images along with the merchandiser-generated annotations or tags on their social media platforms.
As an alternative to opening the app and capturing the image “through” the app, shoppers can open their cameras without using the app and capture the image. Shoppers may then open the apparatus and access desired photos from their photo library immediately after capturing the image or anytime afterward. Because photos are normally geotagged, the apparatus may be set up so that it automatically performs a search of images uploaded by the identified merchandiser, and when a tag for an item remains available, it will be displayed along with the shopper's photo when the shopper accesses the photo from his or her photo library. If the photo is accessed and cannot be matched or annotated, there is no result and no reward. If the annotation or the reward has expired, there is also no result other than the shopper seeing the photo on his or her screen, although the apparatus may include an explanation of the absence of the annotation, such as “Image unrecognized,” or “Information expired,” or “Please go to restaurant website for possible current promotions.” However, if the photo is matched and annotated (tagged), it can be posted on social media platforms along with its current tag; the shopper then receives the current reward from the merchandising venue.
In this embodiment, a shopper might open the apparatus, take a photo of an article of clothing (shirt) at a clothing retailer (J. Crew) and share the tagged version of the photo on his or her social media platform or platforms. The apparatus uses the shopper's CI Device's location means to identify the retailer. The apparatus performs image matching on the shirt by data scraping the images of merchandise uploaded by the retailer, and also finds the retailer's annotation or tag by using an electronic look-up table. Because the retailer has annotated the shirt, the customer's image of the shirt is also annotated—in this example, possibly with “Heavyweight chamois shirt, J. Crew, Madison Avenue, N Y,” followed by a feature such as an icon, a symbol such as a QR code, or a hashtag. For those who will use muses or glide pads, in-image methods may be available, such as moving a mouse over the image to reveal an overlay of a segment of the image or to produce a pop-up box, or to click on the image, thereby directing the user to a new page. Upon posting the image with its annotation or tag, the shopper receives a message about a reward from the retailer via the app—J. Crew—that can say something like “Chamois shirt available for 15% discount until Monday.”
Now consider the experience of a viewer of the image—perhaps a “friend” of the viewer or a friend of a friend—and the annotation posted on the social media platform. The viewer sees the merchandise (shirt) and the tag on the shopper's social media platform. The friend touches the icon, uses an in-image method, or shoots the QR code and obtains supplemental information about the merchandise—in this example, perhaps, “Heavyweight chamois shirt available from J. Crew in green, pale gold, fire, and navy. Regular or slim, sizes x-small to x-large. Touch or click on this link for more information and for a discount.” The retailer builds customer loyalty by rewarding the customer in the store and also, in effect, advertises and markets its wares on shopper's social media platforms. Moreover, rewards for the shopper can accrue over time as viewers of the photos on the social media platforms respond to the photos by contacting or shopping with the merchandiser. That is, shoppers posting tagged photos become, in effect, advertisers or sales people for the merchandiser. In addition, the system may be set up so that viewers of the posted photos and the associated information can obtain rewards by sharing the photos with others who act upon the photo and the information.
If the shopper took a photo of the shirt without using the apparatus, the shopper can later open the apparatus and access the photo of the shirt from his or her photo library. Accessing the photo via the apparatus will automatically tag it with the merchandiser's current tag for the item, if such a tag still exists. The shopper can share the tagged photo and be notified of a reward upon accessing the photo, if rewards have not expired. Again, the shoppers' “friends” on the social media platforms, and the friends of friends, will be able to see the photo and the tag, which identifies the item and the merchandiser. If “perks” still exist for viewers of the item on social media platforms, a feature such as an icon, an in-image method, a symbol such as a QR code, or a hashtag can be linked to the details.
In a related embodiment, the method of rewarding shoppers for posting photos of food dishes is used by food venues such as restaurants. The food venues and their customers (aka diners) might register with the general merchandising app, as described previously, or with a specialized food app that has similar methodology and functions. As with the clothing retailer, registration enables food venues to advertise and market their food dishes on social media platforms and to build customer loyalty through providing diners with rewards for taking photos of their food dishes and posting them along with venues' annotations on their social media platforms.
The apparatus uses the diner's web-enabled CI Device's location means to identify the food venue. The apparatus also engages in image matching to match the image of the food dish with images of food dishes that have been uploaded to an apparatus database by the food venue and annotated or tagged by the food venue. The photo may either be taken “through” the apparatus or accessed by the diner from his or her photo library afterward. Once successful image matching has taken place, the apparatus annotates or tags the photo with a venue-generated message, such as name of the dish, name of the food venue, and food venue location; for example, “Spanakopita, Athens Kouzina, 7th Avenue, N.Y.” With this tag, the photo becomes an advertisement for the food venue. The message may also contain a humanly recognizable or machine-readable feature that is linked to supplemental information and indicates that a reward is available to viewers of the photo on the social media platform, as, for example, a discount on the dish or a general discount. The feature may be, as examples, an icon, a hot spot, a symbol such as a QR code (which would likely be best accessed from the app's platform and displayed on the diner's computer screen), an in-image method, or a hashtag. Touching the icon or hot spot may launch a page with, for example, a QR code and explanatory text that makes viewers aware of a reward that awaits them if they dine at the food venue. The code may be on a screen that the viewer shows at the food venue or printed out. At the food venue, the code can be scanned to complete the reward transaction. Food venues, like other merchandising venues, are able to add or update photos and tags. Viewers of the photo and tag on social media platforms may access the supplemental information by means such as touching the photo or an icon associated with the photo.
The food venue, through the apparatus, may automatically post the image and its annotation to the user's social media platform. Alternatively, after diners take photos, the apparatus may display the social media platforms that had been registered by the diners, permitting the diner to select the platform. The apparatus may also display the image and the tag on the diner's display, permitting the diner to choose whether and where to post the image and the tag. The apparatus posts tagged photos to diners' selected social media platforms and notifies diners of rewards. Said rewards may be delivered via MMS or SMS message, email, or other means.
In practice, the food venue or restaurant engages in steps such as the following: (i) The restaurant registers with the app; (ii) The restaurant uploads images of its food dishes to the database of or associated with the apparatus. There may be multiple photos per dish, taken from different perspectives, or, for example, a meat dish with different sets of vegetables. A tag having more information regarding the dish is associated with each food dish. Photos of food dishes may be deleted and added by the venue. Tags for existing and new dishes may be added by the food venue, and tags may be modified. The app or operators of the app may assist restaurants in the formatting and wording of tags, and in the use of humanly recognizable or machine-readable features that comprise links to supplemental information, such as advertising and marketing material; (iii) The restaurant uploads information about rewards associated with photos of food dishes to the database. The rewards may be specific to the food dish or general. Rewards may be updated by the food venue. Rewards are stored by the apparatus's database, and they are associated with the images of the food dishes. Examples of rewards include discounts, points toward free desserts or entrees, and V.I.P. status—meaning, for example, that diners may be able to “go to the head of the line” in making reservations; (iv) The restaurant provides its location to the app, assisting in geotagging of the images such that diners who take photos in the restaurant and post them later “through” the app are more likely to associate them with the restaurant; (v) The apparatus uses image matching to match diners' images to images uploaded by the restaurant; (vi) The apparatus attaches the associated tags to diners' images and posts the images and their tags to the social media platforms identified by diners; (vii) Alternatively, the restaurant the apparatus may provide users with an intermediate step in which it displays the photo and its annotation on the user's display. The apparatus may then request permission to post the image and tag on the user's social media platform. Users may have the opportunity to press a button or a hot spot to post the image and annotation. Users may change social media platforms, not post at all (that is, cancel the process), save the image, save the image along with the restaurants' annotations, or add their own comments to the image and restaurants' annotations; (vii) The apparatus transmits rewards to diners who take photos of food dishes and arrange for posting them along with restaurants' tags.
In practice, the diner participates in steps such as the following: (i) The diner downloads and registers with an application (or app) to engage the apparatus. Registration may involve steps such as providing the apparatus with permission to post photos to the social media platforms registered by the diner, preselecting at least one social media platform on which the apparatus may post the photo along with the restaurant's annotations, providing the apparatus with permission to access the diner's device's locating means, providing the apparatus with permission to use the diner's device's camera, providing the apparatus with the diner's device's phone number and/or the diner's email address so that the apparatus can transmit rewards to the diner; (ii) Alternatively, the diner may be able to register with the app via one of his or her social media platforms; (iii) After registration, the diner opens the app, which automatically opens his or her device's camera; (iv) The diner takes a photo of a food dish at the restaurant, and the apparatus automatically annotates or tags the image with restaurant-generated information; (v) Alternatively, the diner opens the app and accesses a photo from his or her photo library; accessing the photo after the app is opened results in the app's attempt to automatically tag of the photo with the current available tag from the restaurant; (vi) The app displays social media platforms registered by the diner so that the diner can select the platform on which the tagged photo will be posted; (vii) The diner posts the tagged photo on the social media platform or platforms; (viii) Alternatively, the diner may be afforded the opportunities to save the image without the restaurant's tag, to save the image along with the tag, to edit the image, to add his or her own comment to the photo and tag, and to post the image without the restaurant's tag; (ix) When the diner posts the photo along with the restaurant's tag, whether or not the photo and tag are supplemented with the diner's own message, the diner may be directed to a rewards screen; alternatively the diner receives a personal, printed, or electronic message specifying the reward.
The implementation and support of the apparatus may be accomplished in various ways: Users (shoppers and diners) may pay to download the app and to maintain membership. Merchandising venues (including restaurants) may pay to use the apparatus. Users and merchandising venues may agree that the apparatus can insert third-party advertisements to accompany images and annotations. In the third case, the service is free to users and merchandising venues.
The system need not be used only in the Internet context. Various organizations can use the system to identify people or acquire important information. The database could be maintained by the organizations and contain data such as the image representation of an individual and their desired descriptions. In the medical field, the database can be maintained by medical facilities and entered by medical personnel as the individuals' medical records change. For example, individuals having a chronic illness may contain information identifying their illness and ways of assisting such an individual. In the instance that they have a recurrence, anyone authorized to access the system, such as medical personnel, may capture their image and acquire password protected medical information about the affected person through available wireless Internet based device. This would provide a more secure environment for the sickly, elderly, or the like.
The system can also be used in a secure environment such as, but not limited to a prison, airport, secret agencies, army, hospital, and the like. In these applications, the individual's information includes criminal, immigration, medical records, or the like. Anyone that has access to the system can enter the information about the person of interest into the system. The authorized person can access this personal information via a password or the like. The information can be used to run background checks, to identify individuals in need of help, to find missing individuals, or the like.
The system can be used as a child-loss-prevention system wherein parents or school officials may enter the child's photograph along with identifying information. The identifying information may include the child's name, names and contact information of parents or school officials, and/or the address of the family's residence. For instance, if a user of the system finds a lost child, that user can use the CI device to capture a photograph of the child and acquire the identifying information regarding the child.
The present invention also has great potential for the field of education. The present invention provides a system and method for accessing information regarding an object of question. The source of information retrieved can be books, dictionaries, encyclopedias, articles, news, or the like.
The present invention has further educational potential. For example, the user can capture images of exhibited art or artifacts and enter the images into the image search engine to acquire historic or other information.
An example of a virtual travel guide is shown in
The acquired information is displayed on display screen 1314 of CI device 1300. If captured image 1301 is of a historic building, the information can include the name of the building and a historic profile 1310, including the building's dimensions, the building establishment date, the past usage of the building, and the like. The user of the system enters an inquiry date into the CI device in order to extract information associated with the image on that particular date or thereabouts. The information is extracted from a preset timeline of events, or is used as a search entity to search the Internet. Additionally, date and time 1311 of the captured image is used to extract information associated with that time and the date. For example, the time and date might be associated with information as to whether or not the building is open to visitors. The current time and date coned to be used to extract current news involving building 1312. The database also might search the Internet for available new information posted on that specific time and date and display sites 1313 on the CI device.
Additionally, the system may be used as a translation or dictionary guide to translate signs or written documents. For example, the system captures an image of a street sign in a language foreign to the user and further uses GPS coordinates for assistance to determine the country in which the image is captured. The system further performs image processing to identify the written characters and input the written or printed word or phrase into an electronic translator.
This is a continuation in part of U.S. patent application Ser. No. 14/550,662, filed Nov. 21, 2014, which is a continuation in part of U.S. patent application Ser. No. 14/448,816, filed Jul. 31, 2014, which is a continuation in part of U.S. patent application Ser. No. 14/260,806 filed Apr. 27, 2014, which is a continuation in part of U.S. patent application Ser. No. 14/132,359, filed Dec. 18, 2013, which is a continuation of U.S. patent application Ser. No. 14/083,864, filed Nov. 19, 2013, which is a continuation application of U.S. patent application Ser. No. 13/939,569, filed Jul. 11, 2013, which is a continuation application of U.S. patent application Ser. No. 13/237,849, filed Sep. 20, 2011 and issued as U.S. Pat. No. 8,510,337 on Aug. 13, 2013, which is a continuation application of U.S. patent application Ser. No. 12/799,532, filed Apr. 27, 2010 and issued as U.S. Pat. No. 8,024,359 on Sep. 20, 2011, which is a continuation application of U.S. patent application Ser. No. 11/101,716, filed Apr. 8, 2005 and issued as U.S. Pat. No. 7,765,231 on Jul. 27, 2010, the entire contents of the entire chain of applications is herein incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14083864 | Nov 2013 | US |
| Child | 14132359 | US | |
| Parent | 13939569 | Jul 2013 | US |
| Child | 14083864 | US | |
| Parent | 13237849 | Sep 2011 | US |
| Child | 13939569 | US | |
| Parent | 12799532 | Apr 2010 | US |
| Child | 13237849 | US | |
| Parent | 11101716 | Apr 2005 | US |
| Child | 12799532 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14550662 | Nov 2014 | US |
| Child | 15389037 | US | |
| Parent | 14448816 | Jul 2014 | US |
| Child | 14550662 | US | |
| Parent | 14260806 | Apr 2014 | US |
| Child | 14448816 | US | |
| Parent | 14132359 | Dec 2013 | US |
| Child | 14260806 | US |
