The present disclosure relates to an information processing apparatus, an information processing method, a program, and an imaging apparatus that are capable of combining a plurality of images.
In the past, a technique of combining images shot by a digital camera or the like to generate a panoramic image has been known. Such a panoramic image generation technique is disclosed in Japanese Patent Application Laid-open Nos. 2004-135230 (hereinafter, referred to as Patent Document 1) and 2005-217785 (hereinafter, referred to as Patent Document 2).
The techniques disclosed in Patent Documents 1 and 2 above reduce a user burden caused when a plurality of images to be combined to generate a panoramic image are selected. Such a technique that reduces a user burden and can generate a panoramic image with excellent operability has been demanded.
In view of the circumstances as described above, it is desirable to provide an information processing apparatus, an information processing method, a program, and an imaging apparatus that are capable of generating a combined image such as a panoramic image with excellent operability.
According to an embodiment of the present disclosure, there is provided an information processing apparatus including a calculation unit, a setting unit, a retrieval unit, an arrangement unit, a determination unit, and a notification unit.
The calculation unit is configured to calculate a display range of an input image.
The setting unit is configured to set an inclusive range including at least a part of the calculated display range.
The retrieval unit is configured to retrieve an image to be combined that is associated with the input image.
The arrangement unit is configured to arrange the retrieved image to be combined in the inclusive range.
The determination unit is configured to determine an area within the inclusive range, in which the image to be combined is not arranged, as an image-missing area.
The notification unit is configured to notify a user of information on the determined image-missing area.
In the information processing apparatus, the display range of the input image is calculated, and the inclusive range including at least a part of the display range is set. Then, the image associated with the input image is retrieved as an image to be combined, and arranged in the inclusive range. At this time, the area in which the image to be combined is not arranged is determined as an image-missing area, and the user is notified of information on the area. Therefore, it is possible to easily prepare an image to be allocated to the image-missing area based on the notification, and by combination of those images, it is possible to generate a combined image such as a panoramic image with excellent operability.
The arrangement unit may arrange the image to be combined in the inclusive range based on relevance with the input image.
Accordingly, a combined image can be generated with high accuracy.
The notification unit may notify the user of the determined image-missing area in a visualized manner.
Accordingly, the image-missing area can be visually recognized.
The notification unit may notify the user of support information including at least information on a shooting position and a shooting direction, the support information being used for capturing an image to be allocated to the determined image-missing area.
Accordingly, the image to be allocated to the image-missing area can be easily captured based on the support information.
The information processing apparatus may further include a generation unit configured to generate an interpolation image for interpolating the image-missing area.
In such a manner, the combined image may be generated using the interpolation image to interpolate the image-missing area.
The information processing apparatus may further include a connection unit configured to be connectable via a network to a different information processing apparatus storing one or more images. In this case, the retrieval unit may retrieve via the network the image to be combined from the one or more images stored in the different information processing apparatus.
In such a manner, the image to be combined may be retrieved from the different information processing apparatus via the network. Accordingly, a more appropriate image as an image to be combined can be retrieved from many images.
According to another embodiment of the present disclosure, there is provided an information processing method including calculating, by a calculation unit, a display range of an input image.
By a setting unit, an inclusive range including at least a part of the calculated display range is set.
By the retrieval unit, an image to be combined that is associated with the input image is retrieved.
By the arrangement unit, the retrieved image to be combined is arranged in the inclusive range.
By the determination unit, an area within the inclusive range, in which the image to be combined is not arranged, is determined as an image-missing area.
By the notification unit, a user is notified of information on the determined image-missing area.
According to another embodiment of the present disclosure, there is provided a program causing a computer to function as a calculation unit, a setting unit, a retrieval unit, an arrangement unit, a determination unit, and a notification unit.
The calculation unit is configured to calculate a display range of an input image.
The setting unit is configured to set an inclusive range including at least a part of the calculated display range.
The retrieval unit is configured to retrieve an image to be combined that is associated with the input image.
The arrangement unit is configured to arrange the retrieved image to be combined in the inclusive range.
The determination unit is configured to determine an area within the inclusive range, in which the image to be combined is not arranged, as an image-missing area.
The notification unit is configured to notify a user of information on the determined image-missing area.
According to another embodiment of the present disclosure, there is provided an imaging apparatus including an imaging unit, a calculation unit, a setting unit, a retrieval unit, an arrangement unit, a determination unit, and a notification unit.
The imaging unit is configured to capture an image.
The calculation unit is configured to calculate a shooting range of the captured image.
The setting unit is configured to set an inclusive range including at least a part of the calculated shooting range.
The retrieval unit is configured to retrieve an image to be combined that is associated with the captured image.
The arrangement unit is configured to arrange the retrieved image to be combined in the inclusive range.
The determination unit is configured to determine an area within the inclusive range, in which the image to be combined is not arranged, as an image-missing area.
The notification unit is configured to notify a user of information on the determined image-missing area.
As described above, according to the present disclosure, a combined image such as a panoramic image can be generated with excellent operability.
These and other objects, features and advantages of the present disclosure will become more apparent in light of the following detailed description of best mode embodiments thereof, as illustrated in the accompanying drawings.
Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.
The network 10 is a network using a standard protocol of TCP/IP (Transmission Control Protocol/Internet Protocol) or the like, such as the Internet, WAN (Wide Area Network), WWAN (Wireless WAN), LAN (Local Area Network), WLAN (Wireless LAN), or a home network.
The imaging apparatus 200 includes a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a flash memory 203, a display 204, a touch panel 205, a communication unit 206, an external interface (I/F) 207, and a key/switch unit 208. Further, the imaging apparatus 200 includes an imaging unit 209, a GPS (Global Positioning System) module 210, and an orientation sensor 211.
The CPU 201 exchanges signals with the blocks of the imaging apparatus 200 to perform various computations, and collectively controls processing executed by the imaging apparatus 200, such as imaging processing for images, processing of generating an imaging file including metadata, and the like.
The RAM 202 is used as a work area of the CPU 201 and temporarily stores various types of data processed by the CPU 201, such as captured images and metadata, and programs such as applications.
The flash memory 203 is, for example, a NAND-type flash memory and stores data necessary for various types of processing, content data such as shot images, and various programs such as applications and control programs executed by the CPU 201. Further, when an application is executed, the flash memory 203 reads out various types of data necessary for the execution of the application to the RAM 202.
The various programs described above may be stored in other recording media (not shown) such as memory cards. Further, the imaging apparatus 200 may include an HDD (Hard Disk Drive) or the like as a storage apparatus provided in place of the flash memory 203 or an additional storage apparatus.
The display 204 is an LCD (Liquid Crystal Display), an GELD (Organic Electro-Luminescence Display), or the like. On the display 204, captured images, thumbnail images thereof, or through-the-lens images to be shot are displayed, for example. Further, GUIs (Graphical User Interfaces) for setting shooting conditions or the like, GUIs for using applications or the like, etc. are displayed on the display 204.
As shown in
The communication unit 206 is an interface for connecting the imaging apparatus 200 to the network 10 described above such as WAN (WWAN), Ethernet (registered trademark), and LAN (WLAN) in conformity with the respective standards. The communication unit 206 includes a built-in module for the connection to the WWAN, for example, but it may function when other communication device such as a PC card is attached thereto. The communication unit 206 can switch a connection function with respect to WWAN and WLAN in accordance with an operation of the user so that an active or non-active state is set.
The external I/F 207 is an interface for connecting to external equipment based on the standards such as USB (Universal Serial Bus) and HDMI (High-Definition Multimedia Interface). By the external I/F 207, various types of data such as image files can be transmitted and received to and from the external equipment. Further, the external I/F 207 may be an interface for connecting to various memory cards such as memory sticks.
The key/switch unit 208 receives a user's operation made, for example, with use of a power switch or a shortcut key, which is incapable of being input through the touch panel 205 particularly, and transmits an input signal to the CPU 201.
The imaging unit 209 includes an imaging controller, an image pickup device, and an imaging optical system (not shown). As the image pickup device, for example, a CMOS (Complementary Metal Oxide Semiconductor) and CCD (Charge Coupled Device) sensors are used. The imaging optical system forms an image of a subject on an imaging surface of the image pickup device. The imaging controller drives the image pickup device based on an instruction from the CPU 201 and performs signal processing on image signals output from the image pickup device. Further, the imaging controller controls the imaging optical system to set a zoom magnification of an image to be captured.
The data of the captured image is compressed by a compression system such as JPEG (Joint Photographic Experts Group), and then stored in the RAM 202 or the flash memory 203 or transferred to other equipment via the external I/F 207. Further, in this embodiment, metadata (additional information) defined by Exif (Exchangeable Image File Format) is added to image data. In other words, an image file obtained by adding metadata to image data is transmitted to the server 300 via the network 10.
The GPS module 210 is configured to calculate shooting position information based on a GPS signal received by a GPS antenna (not shown) and outputs the calculated shooting position information to the CPU 201. The calculated shooting position information includes various types of data on a shooting position, such as latitude, longitude, and altitude. It should be noted that as a method of acquiring the shooting position information, other methods may be used. For example, based on information of access points through a wireless LAN present in the surroundings, the shooting position information may be derived. Further, altitude information included in the shooting position information may be generated by, for example, providing a barometer (not shown) to the imaging apparatus 200 and measuring an altitude with the barometer.
In addition, for example, the imaging apparatus 200 may be provided with an angular velocity sensor such as gyroscope, an acceleration sensor, or the like (not shown), and the shooting position information may be acquired with the sensors above. For example, there is a case where a GPS signal is difficult to be received in a canyon between high-rise buildings or the like and position information is incapable of being calculated. In such a case, a displacement from a position where position information can be calculated by the GPS module 210 is calculated with an angular velocity sensor or the like. In such a manner, information of a position where a GPS signal is difficult to be received may be acquired.
The orientation sensor 211 is a sensor that determines an orientation on the earth with use of the earth's magnetism, and outputs a determined orientation to the CPU 201. For example, the orientation sensor 211 is a magnetic field sensor including two coils each having an axis, the axes being orthogonal to each other, and an MR element (magnetoresistive element) disposed at the center of the coils. The MR element is an element that senses the earth's magnetism and has a resistance value varied depending on the strength of the magnetism. A resistance change of the MR element is divided into two directional components by the two coils, and the orientation is calculated based on the ratio between the two directional components of the earth's magnetism.
In this embodiment, the orientation in a shooting direction of the imaging apparatus 200 is determined by the orientation sensor 211. The shooting direction is a direction extending from a shooting position (for example, position where the imaging apparatus 200 is located) to a position where a subject within a captured image generated by the imaging unit 209 is located. Specifically, a direction from the shooting position to a subject located at the center of the captured image is calculated as a shooting direction. In other words, a determined imaging direction corresponds to an optical axis direction of the imaging unit 209. It should be noted that as a method of acquiring shooting direction information, other methods may be used. For example, based on the GPS signal described above, the shooting direction information may be acquired. Further, an orientation magnetic needle having a different structure, or the like may be used as an orientation sensor.
As the imaging apparatus 200, various cameras such as a compact digital camera and a digital single-lens reflex camera, mobile phones, smartphones, various types of PDAs (Personal Digital Assistants) having an imaging function, and the like are used.
As shown in
The CPU 301 accesses the RAM 302 or the like as appropriate and collectively controls the whole blocks of the server 300 while performing various computations. The ROM 303 is a nonvolatile memory in which an OS (Operating System) to be executed by the CPU 301 and firmware including programs, various parameters, and the like are fixedly stored. The RAM 302 is used as a work area or the like of the CPU 301 and temporarily stores the OS, various applications being executed, and various types of data being processed.
A communication unit 306, a display unit 307, an input unit 308, a storage 309, a drive unit 310, and the like are connected to the input and output interface 304. The display unit 307 is a display device using an LCD, an OELD, a CRT (Cathode Ray Tube), or the like. The display unit 307 may be incorporated in the server 300 or may be externally connected to the server 300.
The input unit 308 is a pointing device such as a mouse, a keyboard, a touch panel, or other operation apparatus, for example. In the case where the input unit 308 includes a touch panel, the touch panel may be integrated with the display unit 307.
The storage 309 is, for example, an HDD, a flash memory, or a nonvolatile memory such as other solid-state memory. The storage 309 stores the OS, various applications, and various types of data. In this embodiment, the storage 309 also stores a program such as an application for controlling panoramic image generation processing to be described later. Further, the storage 309 stores, as a database, one or more image data items to be retrieved in the panoramic image generation processing.
The drive unit 310 is a device capable of driving a removable recording medium 311 such as an optical recording medium, a floppy (registered trademark) disk, a magnetic recording tape, or a flash memory. In contrast to this, the storage 309 is used as a built-in device of the server 300 that mainly drives non-removable recording media in many cases. Application programs or the like may be read from the removable recording medium 311 by the drive unit 310.
The communication unit 306 is a modem, a router, or other communication equipment that is capable of connecting to the network 10 and is used for communicating with other devices. The communication unit 306 may perform wired or wireless communication. The communication unit 306 may be used separately from the server 300.
Through the communication unit 306, the server 300 can be connected to the imaging apparatus 200 and another server 400 via the network 10. In other words, the communication unit 306 functions as a connection unit according to this embodiment.
The server 400 also has substantially the same configuration as the hardware configuration as shown in
A position information acquisition unit 212, a zoom magnification acquisition unit 213, and a shooting direction acquisition unit 214 shown in
The zoom magnification described above is calculated based on, for example, a ratio of a focal length at a time of photo shooting to a focal length of 50 mm (35 mm format) of a normal lens. For example, when an image is shot in a focal length of 100 mm, a zoom magnification is doubled. However, a method of calculating a zoom magnification is not limited to the above method. For example, a zoom magnification may be calculated later based on the focal length information acquired by the image analysis unit 215. When a digital still camera or the like is used for performing photo shooting, a focal length is recorded in many cases. With use of this focal length, the zoom magnification information may be acquired.
The image data obtained by the imaging unit 209 is stored in the flash memory 203 and output to the image information transmission unit 216. Information items of a shooting position, a zoom magnification, and a shooting direction and other metadata are also output to the image information transmission unit 216. Then, an image file obtained by adding metadata to each image data item is generated and transmitted to the server 300 via the network 10.
An area-missing image (support information) reception unit 217 shown in
The server 300 includes an information acquisition unit 312, a shooting range calculation unit 313, an inclusive range setting unit 314, an image retrieval unit 315, an image collection unit 316, and an image combination unit 317.
The information acquisition unit 312 acquires metadata such as image information, shooting position information, shooting direction information, and zoom magnification information from the image file transmitted from the imaging apparatus 200. The shooting range calculation unit 313 calculates a shooting range that is a display range of the input image based on the shooting position information and the like. The inclusive range setting unit 314 sets an inclusive range including at least a part of the calculated shooting range.
The image retrieval unit 315 retrieves images to be combined, which are used for generating a panoramic image serving as a combined image obtained based on captured images, from one or more image data items stored in the storage 309 and the like of the server 300. Here, the images to be combined include images combined so as to generate a panoramic image, and combination candidate images to be candidates for the images to be combined.
The image collection unit 316 retrieves images to be combined from one or more image data items stored in another server 400 via the network 10. In this embodiment, the image retrieval unit 315 and the image collection unit 316 function as a retrieval unit for retrieving images to be combined that are associated with the input captured image.
The allocation map generation unit 318 generates an allocation map for allocating an image to be combined to an inclusive range. The allocation unit 319 for images to be combined arranges the images to be combined, which have been retrieved, in the inclusive range based on the generated allocation map. The allocation unit 319 for images to be combined functions as an arrangement unit according to this embodiment.
The missing area determination unit 320 determines an area within the inclusive range, in which an image to be combined is not arranged, as an image-missing area. The interpolation image generation unit 322 generates an interpolation image for interpolating an image-missing area.
The combined image generation unit 321 combines the arranged images to be combined or interpolation images to thereby generate a panoramic image as a combined image.
The area-missing image generation unit 323 and the support information generation unit 324 generate information on the determined image-missing area. The area-missing image generation unit 323 generates an area-missing image in which the image-missing area is visualized. The support information generation unit 324 generates support information that is used for capturing an image allocated to the determined image-missing area and includes at least information of a shooting position and a shooting direction. The area-missing image data and support information are output from the communication unit 306 of the server 300 to the imaging apparatus 200 via the network 10. The area-missing image generation unit 323, the support information generation unit 324, and the communication unit 306 implement a notification unit according to this embodiment.
The operation of the server 300 as the information processing apparatus according to this embodiment will be described.
The image file transmitted from the communication unit 206 of the imaging apparatus 200 via the network 10 is received by the communication unit 306 of the server 300. Accordingly, an image 50 captured by the imaging apparatus 200 is input to the server 300.
Metadata such as image information, position information, direction information, and magnification information is acquired by the information acquisition unit 312 shown in
In this embodiment, latitude and longitude information as shooting position information, shooting direction information, and focal length information included in the camera characteristic information are used for the calculation processing for the shooting range 51 (Steps 201 to 203).
Based on the latitude and longitude information, a shooting position P shown in
Based on the shooting direction information, a shooting direction of the imaging apparatus 200, that is, to which direction from the shooting position P the photo shooting has been performed, is determined. In this embodiment, map information is stored in the storage 309 of the server 300 or the like, and the shooting direction information is represented by north, south, east, and west.
Based on the shooting position information, shooting direction information, and map information, a subject as a shooting target is determined. For example, the map information is referred to so that a building, a natural object, or the like that is present from the shooting position P toward the shooting direction is determined. Then, the center point of the determined subject is set as a feature point Q. It should be noted that instead of the center point of the subject, the position of a distinctive part (for example, gate etc.) of a building or the like to be a subject may be set as a feature point Q.
In this embodiment, an image of Mt. Fuji is used as the captured image 50. Therefore, a point at the center of Mt. Fuji is determined as the feature point Q.
For example, famous buildings and sightseeing spots such as Mt. Fuji and Tokyo Sky Tree may be set in advance as candidates of the feature point Q. In other words, buildings and the like highly possible to be a subject may be set in advance as candidates, and a subject may be selected therefrom. Alternatively, information on a building, a natural object, or the like that is present from the shooting position P toward the shooting direction may be transmitted to the imaging apparatus 200 via the network 10 so that a subject may be set by a user. Accordingly, for example, in the case where a plurality of buildings or the like are determined as candidates of a subject based on the map information, an adequate subject is set.
As shown in
As shown in
The size n of the shooting range is calculated using information items on a size m of an imaging surface S in the X direction, a focal length f, a shooting position P, the position of a feature point Q, and a distance x between the shooting position P and the position of the feature point Q. The information on the size m of the imaging surface S is acquired from the camera characteristic information together with the information on the focal length f. The distance x is calculated based on the latitude and longitude of each of the shooting position P and the feature point Q. It should be noted that in this embodiment, the focal length f is converted in terms of a focal length in the case of using a 35-mm film.
With use of those parameters, the following expression for an angle θ shown in
Based on the result, the size n of the shooting range 51 is represented by the following expression.
Size n of shooting range=Distance x*tan θ*2 [Expression 2]
The size of the shooting range 51 of the captured image 50 in the vertical direction (Y direction shown in
In such a manner, in this embodiment, a shooting range of the captured image is calculated by referring to the map information (Step 204).
As shown in Step 103 of
As shown in
A ratio of a resolution (number of pixels) of the captured image 50 to a resolution (number of pixels) of an assumed panoramic image 60 is calculated. For example, in this embodiment, the size of the captured image 50 is set to a UXGA (UltraXGA) size of 1,600×1,200 (pixel). The size of the panoramic image 60 to be generated is set to 6,700×2,500 (pixel). However, the size of both the images 50 and 60 can be set as appropriate in the present disclosure.
Based on the ratio of the size of the captured image 50 to the size of the panoramic image 60, latitude and longitude information of a point 62 of the inclusive range 61 at the upper left and a point 63 thereof at the lower right is calculated. The latitude and longitude information of the points 62 and 63 is calculated based on the latitude and longitude information of the points 52 and 53 of the captured image 50. Accordingly, the inclusive range 61 as a display range of the panoramic image 60 is set.
The inclusive range 61 may be set such that the captured image 50 is not located at the center thereof. In other words, a relative positional relationship between the captured image 50 and the set inclusive range 61 can be set as appropriate. Further, in this embodiment, although the inclusive range 61 is set so as to include the entire shooting range 51, the inclusive range 61 may be set so as to include not the entire shooting range 51 but at least a part thereof. Alternatively, for example, information items on the shooting range 51 and the inclusive range 61 are transmitted to the imaging apparatus 200 via the network 10. Then, GUIs indicating the shooting range 51 and the inclusive range 61 may be displayed so that the position or the like of the inclusive range 61 may be set by a user.
In this embodiment, the latitude and longitude information of the points of the captured image 50 and the inclusive range 61 at the upper left thereof (point 52, point 62) and the points at the lower right thereof (point 53, point 63) are calculated. However, the information is not limited to that of those points. Latitude and longitude information of a middle point of each of four sides of the captured image 50 and the inclusive range 61 may be calculated, for example.
In Step 104 of
As an image 70 to be combined, an image in which at least a part of a shooting range 71 thereof is included in the inclusive range 61 is retrieved. To retrieve such an image, in Step 301 shown in
The area information is calculated based on the inclusive range 61 set by the inclusive range setting unit 314. For example, as the shooting position information, position information indicating a position substantially the same as the shooting position P of the captured image 50 is calculated. Alternatively, position information of a position in a predetermined range from the shooting position P of the captured image 50 may be calculated. Alternatively, position information indicating a position in the vicinity of a line T connecting the position of the feature point Q (Mt. Fuji) and the shooting position P shown in
The shooting direction information calculated as the area information is calculated based on the latitude and longitude information calculated for the shooting position P of the captured image 50, the point 62 of the inclusive range 61 at the upper left, and the point 63 thereof at the lower right. In addition, a setting method for the area information (position information and direction information), which is used for acquiring an image to be combined whose area within the inclusive range 61 is probably captured, can be set as appropriate.
From the image database stored in the storage 309 and the like of the server 300, an image having position information and direction information matched with or close to the area information described above is acquired (Step 302). Then, from one or more obtained images, a combination candidate image 75 is selected in the processing in Step 303 and the subsequent steps.
In Step 303, it is determined whether a confirmation operation as to whether the obtained image can be adopted as a combination candidate image 75 has been performed for all the images. When the confirmation operation is ended (Yes in Step 303), the image retrieval processing is ended.
In the case where the confirmation operation has been not ended (No in Step 303), a shooting range 71 of an image that has not been subjected to the confirmation is calculated (Step 304). The shooting range 71 may be calculated in the same manner as the calculation processing for the shooting range 51 of the captured image 50, for example.
Whether the calculated shooting range 71 is included in the inclusive range 61 that is to be a panoramic image 60 is determined (Step 305). In the case where the shooting range 71 is included in the inclusive range 61 (Yes in Step 305), the image is adopted as a combination candidate image 75 (Step 306). In the case where the shooting range 71 is not included in the inclusive range 61 (No in Step 305), the image is not adopted as a combination candidate image 75 (Step 307).
In Step 305, a threshold value indicating to what extent the shooting range 71 of a retrieved image is included in the inclusive range 61 may be set so that a combination candidate image 75 may be selected based on the threshold value. The threshold value may be set for the number of pixels of the area included in the inclusive range 61, for example. For example, the number of pixels equal to 10% or less of the number of pixels of the entire panoramic image 60 may be set as a threshold value.
In Step 105 of
This determination is performed when the area information generated in Step 301 of
When it is determined that all the images 70 to be combined have not been acquired (No in Step 105), images 70 to be combined are collected by the image collection unit 316 (Step 106).
Area information of an image that has not been acquired by the image retrieval unit 315 is acquired (Step 401). In other words, area information of images necessary for areas that do not cover the inclusive range 61 is acquired. For example, the area information may be calculated in substantially the same manner as the calculation processing for the area information of images 70 to be combined necessary to the inclusive range 61 (Step 301 of
The calculated area information is transmitted to another server 400 via the network 10. Then, in the server 400, an image having position information and direction information matched with or close to the area information is acquired from the image database stored in a storage and the like. The acquired image is transmitted to the server 300 via the network 10. Accordingly, the server 300 acquires an image associated with the captured image 50 from the server 400 via the network 10 (Step 402).
In Steps 403 to 407, the same processing as that performed in Steps 303 to 307 shown in
In Step 107 shown in
The magnification of the captured image 50 transmitted from the imaging apparatus 200 and that of the combination candidate image 75 are corrected (Step 501). This processing is performed so as to connect the captured image 50 and the combination candidate image 75 to each other to thereby generate a panoramic image 60. Typically, the magnification of the combination candidate image 75 is adjusted to that of the captured image 50.
In the case as shown in
By the allocation map generation unit 318 shown in
The allocation unit 319 for images to be combined allocates the combination candidate images 75 to the respective blocks 81 (Step 503).
In this embodiment, the combination candidate image 75 is first allocated to a block 81 adjacent to the center block 81a of the allocation map 80. For example, as shown in
Next, the captured image 50 and the selected combination candidate image 75 are subjected to matching processing. As shown in
In this embodiment, in the right end area 54 and the left end area 74, luminance gradient information of the respective areas is calculated and a local feature amount referred to as SIFT (Scale Invariant Feature Transform) is calculated based on the gradient information. With use of the local feature amount, the matching processing is performed and a positional relationship between the areas 54 and 74 and a degree of matching therebetween are determined. For example, in the case where a predetermined threshold value is set and the result of matching between the areas 54 and 74 takes the threshold value or larger, the selected combination candidate image 75 is selected again as an image 70 to be combined, which is actually used for combination. Then, the image 70 to be combined is arranged at a position where the areas 54 and 74 are best matched.
It should be noted that in the matching processing for the captured image 50 and the combination candidate image 75, any method may be used. In the case where a local feature amount is used, a method other than the SIFT described above may be used. Further, matching processing in which a local feature amount is not used may be used. For example, matching processing may be performed by calculating a correlation coefficient of each luminance value while relatively moving the areas 54 and 74.
In
In Step 504 of
For example, based on the information of luminance values, an image 70 to be combined, which has a hue closest to that of the captured image 50, is arranged. The hue of the captured image 50 differs due to hours, season, weather, and the like when the captured image 50 is obtained, even when images of the same subject are shot from the same position. Therefore, by the arrangement of the image 70 to be combined that has a close hue, a high-quality panoramic image 60 is generated. An optimum image 70 to be combined may be selected based on the metadata such as a date and time of photo shooting, season, and weather that is attached to each image. Further, for example, high-quality images such as an image without shakes or blurring and an image without noises may be selected as the images 70 to be combined. Accordingly, a high-quality panoramic image 60 is generated.
In Step 503, the missing area determination unit 320 determines an area within the inclusive range 61 in which the image 70 to be combined is not arranged, as an image-missing area. For example, in the case where a combination candidate image 75 having information matched with or close to area information on a block 81 of the allocation map 80 does not exist, the block 81 may be determined as an image-missing area. Alternatively, as a result of the matching processing, a block 81 in which an image 70 to be combined is not arranged may be determined as an image-missing area. By the way, for example, there is a case where even when an image 70 to be combined is arranged in each block 81, an area where an image 70 to be combined is not arranged may be generated depending on a position of each image 70 to be combined. Thus, such an area may be determined as an image-missing area.
When the processing of allocating the combination candidate image 75 to each block 81 is performed, color correction processing is performed on the arranged image 70 to be combined (Step 506). Accordingly, for example, based on the hue of the captured image 50, the hue of the entire panoramic image 60 is adjusted, with the result that a high-quality panoramic image 60 is generated.
In Step 507, the captured image 50 and the images 70 to be combined, arranged in the respective blocks 81, are connected by stitching processing. For example, alignment processing or geometrical conversion processing is performed as appropriate so as not to stand out boundaries between the captured image 50 and the images 70 to be combined. As such stitching processing, processing using the feature amount described above, processing using a correlation coefficient, and any other processing may be performed.
In the case where the entire inclusive range 61 is covered with the images 70 to be combined, which have been arranged to the respective blocks 81, that is, the case where it is determined that image-missing areas do not exist, a high-quality panoramic image 60 as shown in
In the case where the missing area determination unit 320 determines that the image-missing area exists, an area-missing image, support information, and an interpolation image are generated by the area-missing image generation unit 323, the support information generation unit 324, and the interpolation image generation unit 322 shown in
As the area-missing image 90, for example, an image in which the images 70 to be combined that are arranged outside the image-missing area 91 are connected with high accuracy by the stitching processing may be generated. In other words, a high-quality area-missing image 90 having size and resolution substantially the same as those of the panoramic image 60 shown in
Alternatively, as the area-missing image 90, an image having a lower resolution than that of the panoramic image 60 or a thumbnail image may be generated. Further, an image to which the stitching processing is not subjected and in which an image 70 to be combined is arranged to overlap another image 70 to be combined may be generated. In other words, as long as the position of the image-missing area 91 can be recognized by a user, an area having a lower accuracy than that of the panoramic image 60 may be generated as an area-missing image 90. Accordingly, loads on the processing resources such as the CPU 301 and the RAM 302 are reduced and a processing speed can be improved at a certain level.
As shown in
Here, attention is focused on an image-missing area 91a shown in
In this embodiment, the images 70 to be combined and the interpolation images 93 are combined in Steps 506 and 507, with the result that an interpolated panoramic image 95 shown in
The panoramic image 60, the area-missing image 90, the support information, and the interpolated panoramic image are transmitted to the imaging apparatus 200 via the network 10. For example, the area-missing image 90 is displayed on the display 204 of the imaging apparatus 200. Accordingly, a user can visually recognize the image-missing area 91 and easily grasp an image necessary for completing the panoramic image 60. Then, for example, the user can get close to Mt. Fuji to capture an image for covering the image-missing area 91 with the imaging apparatus 200. Alternatively, the user can retrieve and download an appropriate image via the network 10.
In this embodiment, area information (position information and direction information) of the through-the-lens image 250 displayed on the display 204 is acquired. Then, the acquired area information of the through-the-lens image 250 and the support information (position information and direction information) used for covering the image-missing area 91 are compared to each other.
As a result, in the case where the through-the-lens image 250 can be captured as an image for covering the image-missing area 91, an OK mark is displayed as an imaging instruction mark 251 on the display 204. When the imaging instruction mark 251 is displayed, the user presses an imaging button to execute imaging processing. Accordingly, an image for covering the image-missing area 91 can be easily captured with excellent operability. The imaging instruction mark 251 is not limited to the OK mark, and various GUIs may be displayed as the imaging instruction mark 251. Alternatively, an audio to instruct imaging or the like may be issued.
For example, for a period of time during which the through-the-lens image 250 is being displayed on the display 204, the area information of the through-the-lens image 250 may be constantly acquired and compared with the support information. Alternatively, a mode for confirming whether the through-the-lens image 250 is adequate as an image for covering the image-missing area 91 is selected by the user. At that time, the area information of the through-the-lens image 250 may be acquired and compared with the support information.
Further, as shown in
As described above, in the server 300 as the information processing apparatus according to this embodiment, the shooting range 51 as a display range of the captured image 50 that has been input from the imaging apparatus 200 via the network 10 is calculated. Further, the inclusive range 61 including at least a part of the shooting range 51 is set. Then, an image associated with the captured image 50 is retrieved as an image 70 to be combined, and arranged in the inclusive range 61. At this time, an area in which the image 70 to be combined is not arranged is determined as the image-missing area 91, and as information related thereto, the user is notified of the area-missing image 90 and the support information via the network 10. Therefore, based on the notification, an image to be allocated to the image-missing area 91 can easily be prepared, and by combining those images, a combined image such as the panoramic image 60 can be generated with excellent operability.
In this embodiment, by the relevance based on the shooting position information and shooting direction information of the captured image 50, the image 70 to be combined is selected and arranged in the inclusive range 61. Accordingly, the panoramic image 60 can be generated with high accuracy.
Further, in the server 300 according to this embodiment, the area-missing image 90 serving as an image obtained by visualizing the image-missing area 91 is generated. Then, the area-missing image 90 is transmitted to the imaging apparatus 200 via the network 10 and the user is notified of this. Accordingly, the image-missing area 91 can be visually recognized.
Further, in the server 300 according to this embodiment, the user is notified of the support information including at least the information on the shooting position and shooting direction, the support information being used for capturing an image to be allocated to the image-missing area 91. Accordingly, an image to be allocated to the image-missing area 91 can easily be captured.
Further, in the server 300 according to this embodiment, the interpolation image 93 that is interpolated in the image-missing area 91 is generated so that the interpolated panoramic image 95 shown in
Further, the server 300 according to this embodiment is connectable to the server 400 serving as another information processing apparatus storing one or more images via the network 10. Furthermore, the server 300 can retrieve the image 70 to be combined from one or more images stored in the server 400 via the network 10. Accordingly, a more appropriate image as the image 70 to be combined can be retrieved from not only the images stored in the server 300 according to this embodiment but also many images stored in the server 400. As a result, the high-quality panoramic image 60 can be generated.
According to the present disclosure, the high-quality panoramic image 60 can also be generated with excellent operability for a captured image 50 obtained in the past by the captured image 50 being transmitted to the server 300.
The embodiments according to the present disclosure are variously modified without being limited to the embodiment described above.
However, as in a network system 100′ shown in
Further, the imaging apparatus 200 shown in
Similarly, the PC 290 shown in
In the description above, the combination candidate image 75 is retrieved from the storage 309 of the server 300 by the image retrieval unit 315 shown in
In the description above, when an image to be combined (combination candidate image) arranged in the inclusive range is retrieved, the area information (position information and direction information) is referred to. However, tag information on a subject may be added as metadata to a captured image and images stored in each server. Then, images to be combined may be retrieved by referring to the tag information.
For example, when the captured image 50 shown in
It is assumed that photo shooting is performed in a private home of “Mr. Yamada”, for example, and a panoramic image with the captured image as a reference is intended to be generated. In this case, due to the limitation of GPS module accuracy or the fact that information on the home of Mr. Yamada is not acquired from map information, there may be a case where it is difficult to retrieve images showing the inside of his home. In such a case, if tag information indicating “Home of Mr. Yamada” is added to the captured image or images stored in the server, an image to be combined can be easily retrieved by referring to the tag information.
In addition, any data may be used as metadata referred to when an image to be combined is retrieved.
Image processing such as affine transform may be performed on an image to be combined that is arranged in the inclusive range. Accordingly, a shooting direction of a subject can be adjusted and a panoramic image with excellent accuracy can be generated.
In the description above, a panoramic image is generated as a combined image. According to the present disclosure, however, a 3D image may be generated as a combined image. For example, a shooting range of the image captured by the imaging apparatus is calculated. Then, an inclusive range including at least the shooting range is set. The inclusive range is a range in which images to be combined that are necessary for generating a 3D image are arranged and is, for example, a range corresponding to a field of view obtained when a subject of the captured image is viewed from the periphery and above and below. In other words, as an inclusive range, a virtual canvas is set around (side to side and up and down of) the subject. Then, images to be combined that are arranged in the set inclusive range are retrieved by referring to metadata such as position information. Although a 3D image is generated by combination of the retrieved images to be combined, the user is notified of information on image-missing areas where images to be combined are not arranged, at this time. Accordingly, a highly accurate 3D image can be generated with excellent operability.
In the description above, a captured image to which metadata such as position information is added is transmitted to the server. Then, a subject is determined based on the metadata and a feature point is set. However, a subject may be determined without using the metadata. For example, the color, shape, or the like of the captured subject is detected based on luminance information or the like of the captured image. Then, the subject may be determined based on the information on the color, shape, or the like. In addition, various object recognition technologies can be adopted. Accordingly, the present disclosure is also applicable to a captured image to which metadata is not added so that a combined image such as a panoramic image can be generated with excellent operability.
Further, an image to be input is not limited to a captured image. For example, a sketch of Mt. Fuji may be digitized and image data thereof may be input as an input image. Then, a display range of the digital image may be calculated.
As the images to be combined that are arranged in the inclusive range, for example, an image obtained by photo shooting of a large range including a shooting range of a captured image (for example, wide-angle image) may be retrieved. In this case, a panoramic image may be generated using the wide-angle image or the like instead of the captured image. In other words, a panoramic image may be generated without using the captured image. In the case where a resolution of the wide-angle image or the like is insufficient, the resolution may be increased by resolution conversion processing.
In the description above, three of the interpolation image, area-missing image, and support information are generated. However, one of them may be generated. Further, any two of them may be generated.
In the description above, the shooting advice unit 218 is provided to the imaging apparatus 200. However, a block corresponding to the shooting advice unit 218 may be provided to the server 300 that generates the area-missing image 90 and the support information. In other words, the processing performed on the through-the-lens image 250 as shown in
It should be noted that the present disclosure can be configured as follows.
(1) An information processing apparatus, including:
a calculation unit configured to calculate a display range of an input image;
a setting unit configured to set an inclusive range including at least a part of the calculated display range;
a retrieval unit configured to retrieve an image to be combined that is associated with the input image;
an arrangement unit configured to arrange the retrieved image to be combined in the inclusive range;
a determination unit configured to determine an area within the inclusive range, in which the image to be combined is not arranged, as an image-missing area; and
a notification unit configured to notify a user of information on the determined image-missing area.
(2) The information processing apparatus according to Item (1), in which the arrangement unit arranges the image to be combined in the inclusive range based on relevance with the input image.
(3) The information processing apparatus according to Item (1) or (2), in which the notification unit notifies the user of the determined image-missing area in a visualized manner.
(4) The information processing apparatus according to any one of Items (1) to (3), in which the notification unit notifies the user of support information including at least information on a shooting position and a shooting direction, the support information being used for capturing an image to be allocated to the determined image-missing area.
(5) The information processing apparatus according to any one of Items (1) to (4), further including a generation unit configured to generate an interpolation image for interpolating the image-missing area.
(6) The information processing apparatus according to any one of Items (1) to (5), further including a connection unit configured to be connectable via a network to a different information processing apparatus storing one or more images, in which the retrieval unit retrieves via the network the image to be combined from the one or more images stored in the different information processing apparatus.
(7) An information processing method, including:
calculating, by a calculation unit, a display range of an input image;
setting, by a setting unit, an inclusive range including at least a part of the calculated display range;
retrieving, by a retrieval unit, an image to be combined that is associated with the input image;
arranging, by an arrangement unit, the retrieved image to be combined in the inclusive range;
determining, by a determination unit, an area within the inclusive range, in which the image to be combined is not arranged, as an image-missing area; and
notifying, by a notification unit, a user of information on the determined image-missing area.
(8) A program causing a computer to function as:
a calculation unit configured to calculate a display range of an input image;
a setting unit configured to set an inclusive range including at least a part of the calculated display range;
a retrieval unit configured to retrieve an image to be combined that is associated with the input image;
an arrangement unit configured to arrange the retrieved image to be combined in the inclusive range;
a determination unit configured to determine an area within the inclusive range, in which the image to be combined is not arranged, as an image-missing area; and
a notification unit configured to notify a user of information on the determined image-missing area.
(9) An imaging apparatus, including:
an imaging unit configured to capture an image;
a calculation unit configured to calculate a shooting range of the captured image;
a setting unit configured to set an inclusive range including at least a part of the calculated shooting range;
a retrieval unit configured to retrieve an image to be combined that is associated with the captured image;
an arrangement unit configured to arrange the retrieved image to be combined in the inclusive range;
a determination unit configured to determine an area within the inclusive range, in which the image to be combined is not arranged, as an image-missing area; and
a notification unit configured to notify a user of information on the determined image-missing area.
The present disclosure contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2011-019138 filed in the Japan Patent Office on Jan. 31, 2011, the entire content of which is hereby incorporated by reference.
It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.
Number | Date | Country | Kind |
---|---|---|---|
P2011-019138 | Jan 2011 | JP | national |