Patent Application
20090310866
The present invention relates to a technique that superimposes an image or information for advertisement on a moving image or a static image captured using a digital video camera and the like and presents the moving image or the static image having the superimposed image or information to a user.
With the widespread use of digital video cameras (also referred to as “digital cameras”) and mobile phones equipped with function of a camera, the occasion for shooting landscapes, buildings, and the like has been increased in recent years. Along with that, techniques for presenting advertising information or service information to a user by superimposing information related to captured landscapes or buildings on an image and displaying the image having the superimposed information have been proposed.
For example, a system has been proposed which superimposes advertising information and the like suitable to an image captured using a camera mounted on the front end of a railroad vehicle or an airplane on the captured image and displays the image having the superimposed information (see, for example, Patent Reference 1). In the above system, a method for superimposing information obtained from a predetermined server on the captured image and displaying the image having the superimposed information has been disclosed.
In the conventional system, however, information to be displayed is not necessarily suitable to a user's preference since the information to be superimposed on images shot at the front end of a railroad vehicle or an airplane (since routes of these public transportation are predetermined in principle, the images are mostly in a same pattern.) is associated in advance with each of the images. Thus, this poses a problem of limiting an advertising target since information can not be superimposed on an image captured by a user arbitrarily with a user's camera.
The present invention has been conceived in view of the above-described problem and aims to provide a superimposition information presentation apparatus and the like capable of superimposing information suitable to a user's preference on an image captured arbitrarily by the user and presenting the image having the superimposed information.
In order to solve the above-described problem, a superimposition information presentation apparatus according to the present invention superimposes information in association with an image and presents the image having the superimposed information, the apparatus including: an image obtaining unit configured to obtain an image; an analysis unit which analyzes the obtained image and extracts related information that relates to to-be-superimposed information; a superimposition data selection unit which identifies the to-be-superimposed information based on the extracted related information; and a superimposing unit which superimposes the identified information on the obtained image and presents the image having the superimposed information.
This structure makes it possible to superimpose information suitable to a user's preference on an image captured arbitrarily by the user and present the image having the superimposed information.
It should be noted that the present invention can also be implemented as a superimposition information presentation method including, as its steps, the characteristic components included in the superimposition information presentation apparatus. Further, the present invention can be implemented as a program which causes a computer to execute each of the steps of the method described above. Further, it should be understood that such a program can be widely distributed via a recording medium such as a DVD and a transmission medium such as the Internet.
Further, the present invention can also be implemented as: a server apparatus provided with the characteristic components included in the superimposition information presentation apparatus; and a superimposition information presentation system that includes the server apparatus and a terminal apparatus provided with a camera function and a communication function. Further, the present invention can also be implemented as a superimposition information providing method or a superimposition information utilizing method, each of which includes, as its steps, the characteristic components included in the server apparatus or the superimposition information presentation system described above.
With a superimposition information presentation apparatus according to the present invention, it is possible to superimpose information suitable to a user's preference on an image captured arbitrarily by the user and present the image having the superimposed information. Further, it is possible for the user to view a captured image on which information suitable to the user's preference is superimposed automatically, by shooting arbitrary landscape and the like. Further, it is possible for a business organization (advertiser) which provides the information to be superimposed on the image to automatically provide an advertisement suitable to the user's preference. Furthermore, a business organization that provides the superimposition information presentation apparatus can be rewarded from the advertiser for providing advertisement information.
10 superimposition information presentation system
101, 111 superimposition information presentation apparatus
102, 122 server
103 superimposition information
201 recording unit
202 operation input unit
203 interface unit
204 control unit
205 image obtaining unit
206 analysis unit
207 superimposing unit
208 output unit
209 communication control unit
210 sensor unit
301 server information storage unit
302 local information storage unit
303 superimposition data selection unit
304 local information change unit
305 analysis unit
306 superimposition unit
307 communication control unit
308 superimposition data storage unit
801, 804 interface unit
802 superimposition data selection unit
803 portable medium
805 local information storage unit
1003 information registration unit
1004 local information storage unit
1005 analysis unit
1006 superimposition data selection unit
1007 superimposition data storage unit
1008 superimposition unit
Embodiments according to the present invention will be described below with reference to the drawings. It is to be noted that, although the present invention will be described with following embodiments and the drawings, they are intended not for the purpose of limitation but for exemplification only.
The recording unit 201 is, for example, a hard disk device and records information for use in the superimposition information presentation apparatus 101. More specifically, the recording unit 201 records image data of a captured image, information (or, data) to be superimposed, information inputted by a user, and the like. Such data or information is obtained from the image obtaining unit 205, the server 102, or the portable medium 213, or through input by the user, and the like. It is to be noted that the information to be superimposed may include something appealing to the ear (audio), in addition to something appealing to the eye (an image or a text).
The operation input unit 202 is, for example, a keyboard, a mouse, a remote controller, and the like, and receives an operation by the user.
The interface unit 203 is a unit that carries out: detection of the portable medium 213; conversion of data exchanged between the superimposition information presentation apparatus 101 and the portable medium 213; and the like.
The control unit 204 is, for example, a microcomputer including a ROM, a RAM, and the like for storing a control program, and controls an operation of each of the structural components of the superimposition information presentation apparatus 101. Further, the control unit 204 selects information to be superimposed on an image based on local information stored in the recording unit 201. Here, “local information” refers to information that defines a user's preference and characteristics, and is registered in association with a superimposition data ID.
The image obtaining unit 205 is, for example, a digital camera provided with a Charge Coupled Device (CCD), and captures a landscape and the like, converts the captured image into image data, and stores the image data in the recording unit 201.
The analysis unit 206 analyzes the image data obtained by the image obtaining unit 205, extracts a characterizing portion of the image (for example, a two-dimensional code, a character string, or a building), and identify related information that is related to the information to be superimposed, based on the extracted characterizing portion. A specific example for related information includes URL information that indicates a particular site (in other words, capable of identifying a particular server 102)
The superimposing unit 207 superimposes the information selected by the control unit 204 on the image obtained by the image obtaining unit 205.
The output unit 208 is, for example, a liquid crystal display device provided with a speaker, and presents the user with the image on which information is superimposed (further, audio corresponding to the image).
The communication control unit 209 is a part that controls a communication between the server 102 (which is accessed via a network or directly accessed) and the superimposition information presentation apparatus 101. More specifically, the communication control unit 209 obtains, from the server 102, the information to be superimposed.
The sensor unit 210 includes, for example, a clock, a GPS, and the like, and obtains information such as time information, position information of the superimposition information presentation apparatus 101, and weather information. Further, the sensor unit 210 includes a microphone, and receives audio when the above-described image obtaining unit 205 captures image, converts the audio into audio data, and stores the audio data in the recording unit 201. It is to be noted that the above-described image obtaining unit 205 may also include the microphone for receiving audio.
The above-described structure makes it possible that the superimposition information presentation apparatus 101 superimposes to-be-superimposed information received from the communication control unit 209 on an appropriate position of image or audio obtained from the image obtaining unit 205 and presents the image or audio having the superimposed information to a user.
It is to be noted that the image obtaining unit and the sensor unit are examples for obtainment unit.
The present embodiment describes an embodiment that information suitable to a user's preference is superimposed on an image (further, a priority is given to candidate information for superimposition), by using local information held in the superimposition information presentation apparatus 101. The superimposition information presentation apparatus 101 according to the present embodiment, when an image is obtained from the image obtaining unit 205, reads, from the recording unit 201 (more specifically, a local information storage unit 302 in
As illustrated in
The server information storage unit 301 stores superimposition data α1 obtained from the server 102 via the communication control unit 209 and superimposition data management information α2 for managing the superimposition data α1. The superimposition data α1 actually is the information to be superimposed on an image obtained from the image obtaining unit 205. Further, the superimposition data management information α2 defines information related to the superimposition data α1. More specifically, the superimposition data management information α2 defines a genre, a manufacturer, and so on for each superimposition data ID in a table format as illustrated in
The local information storage unit 302 is a part that holds information (local information) related to a user of the superimposition information presentation apparatus 101, and holds: information unique to the superimposition information presentation apparatus 101; a characteristic and preference information of a user obtained via the operation input unit 202; time information; position information; information obtained from the sensor unit 210; and so on. Here, the local information is information having a structure as illustrated in
The superimposition data selection unit 303 is a part that identifies information to be superimposed, according to a user's preference or a situation at the time of shooting. The superimposition data selection unit 303 identifies at least one server 102 that has been identified by the analysis unit 305 and that holds the information to be superimposed, and obtains the superimposition data α1 from the communication control unit 209. In the case where the obtained superimposition data α1 exists as plural items of data, the superimposition data management information α2 and sensor information is obtained respectively from the local information storage unit 302 and the sensor unit 210. A priority is given to the superimposition data α1 by using the obtained information items, so that information to be superimposed is uniquely identified in the end. In the case where the superimposition data management information α2 as illustrated in
The local information update unit 304 updates local information held by the local information storage unit 302 when an operation for updating the local information is received via the operation input unit 202.
The analysis unit 305 is a part that corresponds to the analysis unit 206 in
The superimposing unit 306 is a part that corresponds to the superimposing unit 207 in
The server 102 includes a communication control unit 307 and a superimposition data storage unit 308.
The communication control unit 307 uses the URL information obtained by the analysis unit 305 and accesses the corresponding server apparatus to obtain a candidate of the superimposition information 103.
Further, the communication control unit 307 is a unit that retrieves, from the superimposition data storage unit 306, the superimposition data α1 that has been inquired by the superimposition data selection unit 303, and transfers, to the superimposition information presentation apparatus 101, related information of the retrieved superimposition data α1 and the superimposition data. In the case where Plural items of superimposition data α1 are found as the candidate for a response to the same inquiry, for example, all of the items of superimposition data α1 are transferred. It is to be noted that the communication control unit 307 may obtain the candidate for the superimposition information 103 by holding plural items of URL information in advance and accessing servers each corresponding to one of the plural items of URL information (in other words, by broadcasting).
The superimposition data storage unit 308 is a part that holds plural items of superimposition data α1, and in response to a superimposition data retrieval request from the communication control unit 307, transmits corresponding superimposition data α1.
First, the superimposition data selection unit 303 waits until the analysis unit 305 finishes analyzing an image obtained from the image obtaining unit 205 (Step S601). When data indicating a result of the analysis is inputted from the analysis unit 305, the superimposition data selection unit 303 identifies, using the data, the server 102 from which information to be superimposed is obtained, and transmits a superimposition data obtainment request to the communication control unit 209. In this case, the superimposition data obtainment request is not necessarily limited to one request, but it may be plural requests. The communication control unit 209 communicates with an external server 102 based on the superimposition data obtainment request, and stores the obtained superimposition data α1 and the superimposition data management information α2 in the server information storage unit 301 (Step S602).
The superimposition data selection unit 303, when receiving a notification of completion of obtaining the superimposition data α1 from the communication control unit 209, obtains the superimposition data management information α2 from the server information storage unit 301 and analyzes the details of the superimposition data management information α2 (Step S603). In the case where a result of the analysis shows that the superimposition data α1 exists as plural items of data (Step S603: Yes), the superimposition data selection unit 303 obtains: local information from the local information storage unit 302; and time information, position information, and the like from the sensor unit 210 (Step S604), and gives priority to the superimposition data α1 based on the obtained local information and superimposition data management information α2, thereby selecting information to be superimposed (Step S605).
Further, in the case where only one superimposition data α1 exists (Step S603: No), the superimposition data selection unit 303 notifies the superimposing unit 306 of the selected superimposition data α1.
Selecting to-be-superimposed information based on priority performed by the superimposition data selection unit 303 may be carried out with display on the output unit 208 as illustrated in
When the superimposition data α1 is selected, the superimposition data selection unit 303 notifies the superimposing unit 306 of completion of selecting superimposition data. When receiving the notification, the superimposing unit 306 superimposes the superimposition data α1 obtained from the server information storage unit 301 on an image inputted from the image obtaining unit 205 via the analysis unit 305 (Step S606). A position on which the data is to be superimposed may be determined based on the information inputted from the analysis unit 305, or may be determined based on information that exists in the superimposition data α1.
With the processes described above, it is made possible to superimpose information suitable to a user's preference on an image obtained by the image obtaining unit 205.
The present embodiment describes a superimposition information presentation apparatus 111 that presents information to be superimposed on a image according to a user's preference. The difference between the superimposition information presentation apparatus 111 and the superimposition information presentation apparatus 101 according to the first embodiment described above is a location in which local information is stored. The local information is stored in the superimposition information presentation apparatus 101 in the first embodiment, whereas local information is stored in a portable medium 213 in the second embodiment.
The superimposition information presentation apparatus 111 further includes an interface unit 801, and the control unit 204 includes a superimposition data selection unit 802.
The interface unit 801 is a part that corresponds to the interface unit 203 in
The superimposition data selection unit 802 is almost the same in the function as the superimposition data selection unit 303 in
The portable medium 803 includes an interface unit 804 and a local information storage unit 805. The interface unit 804 is a unit that controls data exchange with the interface unit 801 in the superimposition information presentation apparatus 111. More specifically, local information held in the local information storage unit 805 is transferred as a response when a local information obtainment request is received from the interface unit 801. The local information obtainment request sent from the interface unit 801 to the interface unit 804 can include various requests such as a request for obtaining all of, or only a portion of the local information registered in the portable medium 803.
The local information storage unit 805 holds characteristics or preference information of a user. The local information in the portable medium 803 can be compiled by using an external device such as a Personal Computer (PC), and by operating from the operation input unit 202.
Next, a processing flow of selection of superimposition data α1 and performing superimposition according to the present embodiment will be described. The processing flow is basically the same as the flowchart in
In the case of the present embodiment, local information is obtained from the portable medium 803 via the interface unit 801.
Further, in the case where both of the superimposition information presentation apparatus 111 and the portable medium 803 include the local information storage unit 302 and the local information storage unit 805, respectively, it may be possible: to use all of the registered local information; to provide a display as illustrated in
Whereas image data analysis and selection and superimposition of superimposition data are carried out in the superimposition information presentation apparatus 101 or the superimposition information presentation apparatus 111 in the first embodiment and the second embodiment described above, these processes are carried out in the server 122 in a third embodiment.
The superimposition information presentation apparatus 121 includes a communication control unit 1001.
The communication control unit 1001 transmits: an image inputted from the image obtaining unit 205; sensor information obtained from the sensor unit 210; information unique to the superimposition information presentation apparatus 121; and information related to a user, to a communication control unit 1002 in the server 122. Unlike the first embodiment and the second embodiment described above, the inputted image is transmitted to the server 122 without being analyzed.
The server 122 includes: the communication control unit 1002; a information registration unit 1003; a local information storage unit 1004; an analysis unit 1005; a superimposition data selection unit 1006; a superimposition data storage unit 1007; and a superimposing unit 1008.
The communication control unit 1002 is a unit that controls data exchange with the superimposition information presentation apparatus 121. More specifically, the communication control unit 1002 is a part that obtains image data and user information from the superimposition information presentation apparatus 121. Further, the communication control unit 1002 is a part that transfers data superimposed in the server 122 to the superimposition information presentation apparatus 121.
The information registration unit 1003 is a part that resister user information on the local information storage unit 1004 of the server. A user sets information specific to the user, which includes personal information, preference, and the like, using an external registration device, such as a PC.
The local information storage unit 1004 is a part that hold information specific to the user. More specifically, the local information storage unit 1004 holds a table that associates a user with preference specific to the user, similarly to the local information storage unit 302 that holds local information as illustrated in
The analysis unit 1005 is a part that analyzes an image obtained via the communication control unit 1002. The analysis unit 1005 analyzes an image inputted from the image obtaining unit 205 and extracts a characteristic to be a candidate for information to be superimposed.
The superimposition data selection unit 1006 selects information to be superimposed based on: characteristic information that is the candidate for superimposition data and is inputted from the analysis unit 1005; superimposition data management information α2 held by the superimposition data management unit 1007; and local information specific to the user, which is obtained from the local information storage unit 1004.
The superimposition data storage unit 1007 holds data α1 that is the data to be superimposed and superimposition data management information α2 that is characteristic information of the data α1.
The superimposing unit 1008 is a part that superimposes the selected superimposition data α1 on the image data inputted from the analysis unit 1005, in response to a notification of completing a superimposition data selection process from the superimposition data selection unit 1006.
First, the communication control unit 1002 waits until data is inputted from the communication control unit 1001 (Step S1101).
When data is inputted, the analysis unit 1005 analyzes the data inputted in Step S1101 and analyses a candidate for characteristic information for identifying superimposition data based on the inputted data (Step S1102).
The superimposition data selection unit 1006 identifies superimposition data α1 to be superimposed based on the characteristic information of candidate for superimposition data which has been identified in Step S1102 and superimposition data management information α2. Here, in the case where superimposition data α1 exists as plural items of data (Step S1103: Yes), local information is obtained from the local information storage unit 1004 (Step S1104).
Further, in the case where only one or less item of superimposition data α1 exists, the superimposition data selection unit 1008 is notified of completion of selecting superimposition data (Step S1103: No).
In the case where plural items of superimposition data are obtained, a priority is given to superimposition data based on the obtained local information and the superimposition data management information α2, so that information to be superimposed is selected. Further, this process may be carried out not automatically, but by providing a display on the output unit 208 as illustrated in
When the superimposition data is selected, the superimposing unit 1008 is notified of completion of selecting superimposition data. The superimposing unit 1008, in response to the notification, carries out the processing of superimposing the superimposition data α1 obtained from the superimposition data storage unit 1007 on camera data inputted from the analysis unit 1004 (Step S1106). A position on which the data is to be superimposed may be determined based on the information inputted from the analysis unit 1005, or may be determined based on information that exists in the superimposition data α1.
First, when a target image on which information is to be superimposed and audio or an optical image are obtained via the image obtaining unit 205 and the sensor unit 210 (Step S1401: Yes), the analysis unit 305 analyzes and converts into a signal waveform, the obtained audio or light picture (Step S1402), and compares the signal waveform with the reference waveform within the recording unit 201 (Step S1403). Here, in the case where a matching waveform (or, a waveform having high similarity) exists (Step S1404: Yes), a superimposition data code associated with the reference waveform is identified (Step S1205), and superimposition data is obtained by accessing a server corresponding to the superimposition data code (Step S1406). The superimposing unit 306 superimposes obtained superimposition data on a target image (Step S1407).
With the processes described above, it is made possible to superimpose information suitable to a user's preference on an image inputted from the image obtaining unit 205.
It is to be noted that, although in the first embodiment and the second embodiment, a server from which information to be superimposed is obtained is identified within a network, by analyzing image data inputted via the image obtaining unit 205, information to be superimposed may also be obtained from a local server. For example, by obtaining, from a server that can be connected to only in an amusement park and the like, information to be superimposed, it made possible to obtain information that can be obtained only at the site. As described above, information to be superimposed may be obtained by identifying a shooting location of an image and accessing a site that provides information corresponding to the location.
Further, in the above-described first to third embodiments, it may be possible, by matching analyzed data and data registered on a server, to notify a user of the information. For example, in the case where a wanted criminal suspect has been identified from an analysis of an image of a landscape shot arbitrary by a user, a notification may be displayed on a device the user holds, or a report may automatically be sent to a facility such as a nearby police.
Further, in the above-described first to third embodiments, although the local information is registered by a user, it is not limited to this. The local information may be downloaded from a site and the like, or may be obtained by another user.
Further, in the above-described first to third embodiments, although information is superimposed every time, it may be possible to make a user decide whether or not information should be superimposed.
Further, it may be possible, according to the present invention, to include an audio conversion unit that converts, without displaying, information to be superimposed into audio data, to record the converted audio data on a portable medium and the like, and when reproducing an image, to reproduce audio in association with the image (or, to reproduce audio irrespective of the image).
A superimposition information presentation apparatus according to the present invention is useful in that information suitable to a user's preference can be superimposed on an image shot arbitrarily by the user. Further, it can be applied to other applications such as a surveillance system.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2006-212333 | Aug 2006 | JP | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/JP2007/065078 | 8/1/2007 | WO | 00 | 2/2/2009 |
