Patent Application
20100208997
In contemporary online advertisement platforms, whether or not an advertisement is deemed relevant for a user is largely decided by advanced keyword matching. Typically, in a Pay-Per-Click (PPC) model, advertisers specify the words that are to trigger their advertisements and the maximum amount they are willing to pay per click. When a user enters a search query, browses a web page, or in general interacts with some text, the advertisement platform selects and shows relevant advertisements based on the text content in the query or the page.
Although other contextual information such as location, time, and user profile data may be taken into consideration, textual understanding is still the primary technology in advertisement selection. However, other than text, there is no known mechanism for relating image content to advertisements.
This Summary is provided to introduce a selection of representative concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way that would limit the scope of the claimed subject matter.
Briefly, various aspects of the subject matter described herein are directed towards a technology, such as implemented in a platform that returns advertisements to application servers, by which an input image is image matched against advertiser-provided image-related data to locate a relevant advertisement that applies to the input image. For example, a user may identify an image based upon interaction with web content or may transmit an image, and in response receive one or more advertisements. Advertisers may bid on images and scenes of images to match against input images. A region of interest within an image may be bid and/or matched rather than an entire image; when matching images, different image regions may have different weights according to associated interest maps.
In one aspect, advertiser-provided images are processed into features. The features are associated with advertisements, such as by being used to index advertisements. When an input image is received, features are similarly extracted from the input image, and used (e.g., as an index) to locate one or more relevant advertisements.
In one aspect, advertisers may use a tool to create a new scene based upon an uploaded image, or add the uploaded image to an existing scene. An advertiser may also add context to the image/scene, such as commercial information about the scene.
Other advantages may become apparent from the following detailed description when taken in conjunction with the drawings.
The present invention is illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:
Various aspects of the technology described herein are generally directed towards an image-based advertisement platform, in which advertisers bid on images (instead of or in addition to keywords). In general, this platform applies to scenarios in which images are the main input or consumed content, for example, in content-based image retrieval or Multimedia Messaging Service (MMS) applications. Users receive advertisements based on the content of images that they recently interacted with, e.g., browsed or used. As described below, one implementation of such a platform is based upon image content understanding, image matching, and user understanding; also included is an advertisement editorial tool.
While an advertisement model is described herein, it should be understood that this is only one practical example, and other (e.g., non-commercial) uses for the image-related technology described herein are feasible. Further, billboard images are used herein as one example type of images. However, it should be understood that advertisers may bid on any images that may be easily recognized. As such, the present invention is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the present invention may be used various ways that provide benefits and advantages in computing, image technology and online advertising in general.
In general, advertisers may upload and bid on images 110, instead of (or in addition to) text, and this information is maintained on the advertisement platform server 104 as generally described below. These images are associated with bids and matched to an input image for purpose of returning one or more relevant advertisements. For example, a toy seller may bid on the image of a related movie poster, while a restaurant may bid on an image corresponding to a cooking magazine's cover. Note that an image need not be an entire image, but rather some region thereof. For example, an advertiser may be on a logo within images, so that, for example, whenever a user sends an image with such a logo, a match will be detected. As can be readily appreciated, the same image may thus have different bids associated with different regions of that image, with possibly different advertisers.
To maintain information about an image for subsequent matching purposes, a feature extractor mechanism 112 extracts features from the advertiser images 110, and processes the content into various scenes 114. Noted that while extracting features from an image is a well-known technology, there is no requirement as to any particular set of features that needs to be used, and any existing features (e.g., the well-known Scale-Invariant Feature Transform, or SIFT features) may be used, and/or other known features, as well as features not yet in use.
Further, note that a scene is a set of one or more images conceptually related in some way. For example, consider a user that takes a picture of a billboard, and that picture corresponds to an input image that an advertiser bids on for the purpose of returning a related advertisement. Because of image matching limitations, it is more likely that the input image will be matched to an advertiser's image if the advertiser has a similar image captured from the same general angle as the input image, with similar lighting conditions, and so forth. To improve the chances of matching an input image, the advertiser may choose to provide a scene comprising multiple images, such as various images of the same object taken at different angles and lighting conditions. Note however that different advertisers may bid on different images of the same object or the like, e.g., a helicopter tour service may bid on an aerial view of a coastline, while two resorts may bid on different images of the same coastline each taken from their resort's perspective.
A mechanism 116 adds such advertiser-bid scenes to the advertisement platform server 104. One such mechanism includes a tool that advertisers may use, as generally described below with reference to
With respect to online processing user queries or other communications, the application server 102 communicates with the advertisement platform server 104 to obtain relevant advertisements according to user images 120. As represented in
In general, a matching mechanism 130 of the advertisement platform server 104 returns one or more advertisements based upon a query input image (or part of that image, referred to as a scene). An image may be obtained as a query whenever a user interacts with (e.g., clicks on or hovers over) an image on a web page. In addition to queries that correspond to images related to web-page browsing and the like, a query may be based upon an image that is communicated in some other way. For example, a user may send an image via MMS to another user, and either or both users may receive an advertisement matched to that image by treating that image as a query (when received by the MMS service). A user may also take a picture of something and request related advertising; for example, a user may take a picture of a commercial product, send in the picture, and receive a discount coupon as a returned advertisement, (or information as to how to get such a coupon).
More particularly, as described above, uploaded advertisement images (module 220) are processed into features (block 224) that are maintained in a feature data store 226. Each image or scene within an image is maintained in an image/scene data store 228; note that both data stores may be accessed by the image delete module 221, such as to delete old images. For efficiency, via the index creator module 222, the features may be used as an index to efficiently locate a scene/associated advertisement as referenced in an index data store 230, e.g., the features may correspond to one or more keys.
To online match an image to an advertisement, the framework accepts an input query as an advertisement request 240, which includes the input image and any related context information (described below). The image part of the request 240 similarly has its features extracted (block 242) for matching with the features of the advertisement images. For example, if the features are used as keys to the index data store 230, relevant advertisements may be looked up and ranked based on similarity to the features of the advertisers' images/scenes. To this end, a result ranker 244 selects the closest matching advertisement (or ranked set of advertisements) based on the entries in the index data store 230; note that blocks 242 and 244 of
In general, images are organized into scenes which represent a particular visual object that can be bid by advertisers. As generally represented at step 304, the tool checks whether an uploaded images is a duplicate (that is, a close copy) of an existing scene already in the advertisement platform server. If so, step 306 identifies the duplicate images or scenes to the advertiser, to give the advertiser the opportunity to evaluate the images. For example, this addition may be an error, and although not explicitly shown in
If the advertiser elects to continue, step 308 asks the advertiser whether the image is to be added to an existing scene, or to be used in creating a new scene. If added, step 308 branches to step 310 which represents some manual and/or automated process verifying the image. This ensures that an advertiser does not include an image that may compromise the system in some way. Step 312 then adds the image (assuming verified as acceptable), as well as adding features and indexing data to the appropriate data stores. Step 314 modifies any commercial information associated with the image and/or scene, as described below.
If not a duplicate image at step 302 or if the advertiser has elected to create a new scene at step 308, step 314 is executed to create a new scene for the image. Step 316 verifies the image, which if verified as acceptable, is added to the image data store at step 318. Step 318 also represents as adding the features and indexing data to the appropriate data stores.
Step 320 represents associating commercial information with the scene. This provides context which may help in image matching/advertisement retrieval performance, as well as for other purposes. For example, the corporation name, geographical location of the source of the image (e.g., where the billboard is located), a web URL and interest map data (corresponding to what location within each image is of what interest) may be added.
Step 508 represents locating the advertisement content based upon the match with the input image's features. Step 510 returns the advertisement set.
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote computer storage media including memory storage devices.
With reference to
The computer 610 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer 610 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer 610. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above may also be included within the scope of computer-readable media.
The system memory 630 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 631 and random access memory (RAM) 632. A basic input/output system 633 (BIOS), containing the basic routines that help to transfer information between elements within computer 610, such as during start-up, is typically stored in ROM 631. RAM 632 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 620. By way of example, and not limitation,
The computer 610 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media, described above and illustrated in
The computer 610 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 680. The remote computer 680 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 610, although only a memory storage device 681 has been illustrated in
When used in a LAN networking environment, the computer 610 is connected to the LAN 671 through a network interface or adapter 670. When used in a WAN networking environment, the computer 610 typically includes a modem 672 or other means for establishing communications over the WAN 673, such as the Internet. The modem 672, which may be internal or external, may be connected to the system bus 621 via the user input interface 660 or other appropriate mechanism. A wireless networking component 674 such as comprising an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a WAN or LAN. In a networked environment, program modules depicted relative to the computer 610, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
An auxiliary subsystem 699 (e.g., for auxiliary display of content) may be connected via the user interface 660 to allow data such as program content, system status and event notifications to be provided to the user, even if the main portions of the computer system are in a low power state. The auxiliary subsystem 699 may be connected to the modem 672 and/or network interface 670 to allow communication between these systems while the main processing unit 620 is in a low power state.
While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents failing within the spirit and scope of the invention.
