The present invention relates to an image data storage device, image data storage method, and a non-transitory computer-readable storage medium.
The Moving Picture Experts Group (MPEG) has developed a standard for storing a single still image, a plurality of still images, or an image sequence (e.g., a burst of still images) into a single file. This standard is referred to as High Efficiency Image File Format (HEIF), which enables the interchange, editing, and display of images and image sequences. The HEIF is a storage format extended based on tools provided in international Organization for Standardization (ISO) Base Media File Format (ISOBMFF). HEIF has been standardized to the name of “Image File Format” in ISO/International Electrotechnical Commission (IEC) 23008-12. HEIF specifies a normative structure including metadata, and also specifies a method for associating metadata with an image, and a configuration of metadata in a specific format. PTL1 discusses a technique for storing derivative images into an image file conforming to HEIF.
In contrast, in recent years, image generation devices, such as a camera and a smartphone, have various functions and thus can generate various information, including not only an age capturing date and time, an image size, and an image quality, but also information obtained during image capturing, and metadata corresponding to captured image data. For example, information about a location where image data is captured, information for identifying an object or a scene in image data, and various information associated with image data, such as an image capturing mode used during image capturing, are generated together with image data. These pieces of information about image data can be stored as metadata in an HEIF file.
PTL1: U.S. Patent Laid-Open No. 2016/0371265, Specification
In a case where two or more images of a plurality of images stored in an image file (e.g., an HEIF file) conforming to a predetermined image format correspond to common metadata, it is difficult to effectively perform the processing.
Specifically, even when the same metadata is applied to two or more images stored in the HEIF file, the metadata is stored for each image. Thus, a device that processes the HEIF file cannot determine whether metadata corresponding to each image is common without sequentially checking the metadata corresponding to each image and configuration information about the metadata. In a case where all of a plurality of images in the HEIF file correspond to common metadata, or in a case where two or more of the plurality of images correspond to common metadata, the processing load for collectively performing processing on each image becomes large, accordingly.
For example, in a case where a single image is divided into tiles and each tile image is stored as a derivative image, it is not efficient to define metadata, such as Exchangeable image file format (Exif) data, for each derivative image, from a view point of file generation load and file size. In other words, despite the fact that metadata (e.g., an image size, color information, and encoding information) corresponding to each tile image is common in many cases, metadata has heretofore been defined for each of a plurality of (derivative) images, which is not efficient.
An object of the present invention is to make it possible to effectively perform processing on an image file, in a case where two or more images of a plurality of images stored in the image file conforming to a predetermined file format correspond to common metadata.
To solve the above-described issue, an image data storage device according to the present invention includes, for example, the configuration described below. That is to say, an image data storage device that stores a plurality of images in an image file conforming to an image file format includes an acquisition unit configured to acquire the plurality of images, an identification unit configured to identify metadata common among two or more of the plurality of images acquired by the acquisition unit, a storage unit configured to store the plurality of images acquired by the acquisition unit in the image file conforming to the image file format and to store the metadata identified by the identification unit as common metadata in the image file conforming to the image file format, and an output unit configured to output the image file in which the plurality of images and the metadata are stored by the storage unit.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note, the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but limitation is not made an invention that requires all such features, and multiple such features may be combined as appropriate. Furthermore, in the attached drawings, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.
A first exemplary embodiment of the present invention will be described with reference to the drawings, Not all combinations of features described in the following exemplary embodiments are essential to the present invention.
Programs to be executed by the control unit 106 include an operating system (OS), a driver, and an application. The control unit 106 controls an overall operation of the image data storage device 101, and performs processing for, for example, changing display of the display unit 112, sending an image capturing instruction to the image capturing unit 107, and sending a connection instruction to the LAN control unit 114, based on an instruction input by a user using the operation unit 108. The image capturing unit 107 includes an image sensor, such as a complementary metal-oxide semiconductor (CMOS) sensor, and inputs an image signal according to an instruction from the control unit 106. The encoding unit 111 encodes the input image signal into digital data. The encoding unit 111 also decodes an image file stored in the nonvolatile memory 103. Typically, the image data storage device 101 decodes the image file according to a user operation related to playback processing, and causes the display unit 112 to display the decoded image data.
The metadata processing unit 110 acquires data encoded by the encoding unit 111, and generates an image file conforming to a predetermined file format (e.g., High Efficiency Image File Format (HEIF)). The metadata processing unit 110 can generate not only a file conforming to HEIF, but also a file conforming to, for example, other moving image file formats specified in Moving Picture Experts Group (MPEG), or a format, such as Joint Photographic Experts Group (JPEG). The metadata processing unit 110 can also acquire encoded data from a device other than the encoding unit 111 and generate an image file.
The file output unit 109 outputs the image file generated by the metadata processing unit 110. An output destination of the file output unit 109 is not particularly limited. For example, the image file may be output to a display device that display an image based on the image file, or a printing device that prints an image based on the image file. Alternatively, the image file may be output to a storage device that stores the image file, or a storage medium (nonvolatile memory 103) that stores the image file. The nonvolatile memory 103 is, for example, a Secure Digital (SD) card, CompactFlash®, or a flash memory. The image recognition unit 113 executes recognition processing on, for example, a person, an object, and a scene based on an image signal input from the image capturing unit 107, or an image file stored in the nonvolatile memory 103, and sends the result (scene information or object information) to the control unit 106. The control unit 106 sends an instruction to the display unit 112, or sends an automatic shutter releasing instruction or the like to the image capturing unit 107, based on the recognition processing result. The control unit 106 performs processing for, for example, notifying the metadata processing unit 110 of metadata stored in the image file. The present exemplary embodiment is described assuming that the terms “metadata” and “property information” have substantially the same meaning.
In the present exemplary embodiment, functional blocks are configured using different circuits, but instead processing (operation) of at least one functional block may be implemented by program execution of a central processing unit (CPU) or the like.
The RAM 105 is a main storage unit for the image data storage device 101, and is mainly used as a temporary data storage area when processing of each functional block is executed. The ROM 104 is a nonvolatile storage unit that stores software programs to be executed by each functional block. The programs stored in the ROM 104 are transferred to the RAM 105 and are read out and executed by each functional block. The LAN control unit 114 is a communication interface for connecting to a LAN, and executes communication control using a wired LAN or wireless LAN. For example, in a case where the image data storage device 101 is connected to another device via a wired LAN, the LAN control unit 114 includes transmission media physical (PHY) and media access control (MAC) (transmission media control) hardware circuits. In this case, the LAN control unit 114 corresponds to an Ethernet® Network Interface Card (NIC). In a case where the image data storage device 101 is connected to another device via a wireless LAN, the LAN control unit 114 includes a controller for executing control of a wireless LAN conforming to, for example, Institute of Electrical and Electronics Engineers (IEEE) 802.11a/b/g/n/ac, a radio frequency (RF) circuit, and an antenna.
Next, a processing flow for generating an image file conforming to an image file format (HEIF) will be described with reference to
In step S201, an image signal acquired by the image capturing unit 107 is encoded by the encoding unit 111, and the encoded image signal is input to the metadata processing unit 110. The image data storage device 101 can acquire image signals corresponding to a plurality of images. For example, when the image capturing unit 107 executes burst shooting (continuous shooting), a plurality of continuously captured images is acquired. When an image captured by the image capturing unit 107 is divided into tiles and each tile is encoded by the encoding unit 111, tile images are acquired as a plurality of images. Examples of a method for dividing an image into tiles and encoding each tile include a method using tile coding specified in High Efficiency Video Coding (HEVC), a method for individually encoding divided areas, and other methods, and the method is not limited. The image data storage device 101 can acquire one or more images not only from the image capturing unit 107, but also from other devices.
In step S202, the metadata processing unit 110 analyzes metadata corresponding to the encoded image signal. In step S203, the metadata processing unit 110 acquires metadata corresponding to the image acquired in step S201. A case will be described where the metadata processing unit 110 acquires, for example, an image file conforming to international Organization for Standardization (ISO) Base Media File Format (ISOBMFF) as the encoded image signal. In this case, the metadata processing unit 110 acquires property information to be stored in an item property box (iprp) within the image file, and property information to be referenced by an item in an image property association box.
In the case of acquiring, for example, an image file including Exif data as the encoded image signal, the metadata processing unit 110 acquires Exif data. The data to be acquired is not limited to Exif data. Metadata and the like specified in Extensible Metadata Platform (XMP) and MPEG-7 may be acquired. Alternatively, for example, a part of Exif data may be acquired in step S203.
Image information recognized by the image recognition unit 113 may be used as metadata. For example, the image recognition unit 113 may recognize, the type of a scene for an image, or whether a specific object is included in an image, and may acquire the result (scene recognition result or object information) as metadata, Metadata based on the recognition processing result can also be used as common metadata.
In step S204, the metadata processing unit 110 determines whether one or more images are already present in an HEIF file. If one or more images are present (YES in step S204), the processing proceeds to step S205. If one or more images are not present (NO in step S204), the processing proceeds to step S207. In step S205, the metadata processing unit 110 determines whether the metadata already stored in the HEIF file matches the metadata corresponding to the currently acquired image. If these metadata match (YES in step S205), the processing proceeds to step S206, and the metadata processing unit 110 associates an item ID (image identification information) for the currently acquired image with the common metadata already stored in the HEIF file. In contrast, if the processing proceeds to step S207 from step S204 (NO in step S204), the metadata processing unit 110 associates the item ID for the currently acquired image with the metadata for the currently acquired image, and stores the item ID and the metadata in a new HEIF file. If the processing proceeds to step S207 from step S205 (NO in step S205), the metadata processing unit 110 associates the item ID for the currently acquired image with the metadata for the currently acquired image, and stores the item ID and the metadata in the generated HEIF file.
It may be determined whether to use all or some of the metadata as common metadata based on any setting. For example, only Exif data may be used as common metadata, or only metadata in the image property association box may be used as common metadata. More alternatively, only information about an image capturing date and time may be used as common metadata.
In step S208, the metadata processing unit 110 determines whether there is an unprocessed image. If there is no unprocessed image (YES in step S208), the processing proceeds to step S209. If there is an unprocessed image (NO in step S208), the processing returns to step S201. The determination in step S208 can be made based on, for example, a setting (e.g., the number of images captured by burst shooting) for image capturing conditions, a user's setting for the number of images, and other predetermined conditions.
In step S209, the metadata processing unit 110 deletes, from a storage area for common metadata, metadata that is not common among the two or more images. In the present exemplary embodiment, metadata that is not common among the two or more images is deleted, but instead, for example, a threshold other than “2” may be used to determine whether to delete metadata.
In some cases, metadata corresponding only to a single image may be stored in the storage area for common metadata. This configuration is effective, for example, in a case where only one image is stored in the HEI file, or in a use case where an image item where specific metadata is recorded is to be promptly retrieved. In other words, information stored in the storage area for common metadata can be used as an index for an image associated with specific metadata.
In step S210, the metadata processing unit 110 stores common metadata in the HEIF file. The common metadata may be stored in the HEIF file in step S206 or step S207.
If the storage of common data in the HEIF file eliminates the need for metadata corresponding to each image, the metadata processing unit 110 deletes the metadata from the HEIF file. For example, if the item property association is divided into groups, the item property association for each item is not required and thus is deleted.
Next, a configuration example of a format for the HEIF file generated by the image data storage device 101 will be described with reference to
When a single image is divided into a plurality of tiles and the tiles are stored in the HEIF file, information indicating the association between the tiles is stored. For example, assume that the entire image corresponds to the item 1 and a plurality of tile images corresponds to items 2, 3, 4, and 5. In this case, information indicating that the item 1 is an image formed by the item 2, the item 3, the item 4, and the item 5 is stored. Specifically, “dimg” representing a derivative image is stored as the reference type, and the ID indicating the item 1 is stored in the from_item_id. All the item IDs indicating the item 2, the item 3, the item 4, and the item 5, respectively, are stored in the to_ite_id. With this configuration, information for reconfiguring a plurality of image items, which is obtained by dividing an image into tiles, into one image is indicated.
The ItemReferenceBox can also be used to describe a reference relationship between metadata, such as Exif data, which is defined by external standards, and image items. In this case, “cdsc” is used as the reference type, in the from_item_id, the item ID indicating Exif data is stored, and in the to_item_id, the item ID indicating image item is stored. A box 310 is referred to as an ItemPropertyBox and is identified by an iprp identifier. In the box 310, property information to be applied to each item and a method for configuring the property are stored. A box 311 is referred to as an ImagePropertyContainerBox and is identified by an identifier ipco. In the box 311, a box for describing each property is stored. The box 311 includes various boxes. For example, a box indicating the size of an image, a box indicating color information, a box indicating pixel information, a box for storing HEVC parameters, and the like are stored, as needed. These file formats are common to the box structure specified in ISO/International Electrotechnical Commission (IEC) 23008-12.
A box 312 is referred to as an ItemPropertyAssociationGroupBox and is identified by an ipag identifier. The box 312 is defined by a structure illustrated in
A box 313 is referred to as an ItemPropertyAssociationBox and is identified by an ipma identifier. The box 313 is defined by a structure illustrated in
In contrast, the group bit “1” is set to each item grouped using the ItemPropertyAssociationGroupBox so as to make clear that the property configuration for each item group is described. Further, property configurations to be applied to each group identified by the item_association_group_ID are sequentially described, With this configuration, unlike in the structure of related art in which all item configurations are described for each item, the items to which common properties are applied can be grouped and the item configuration can be described for each group. This leads to a reduction in the amount of data to be described in the file format. In the present exemplary embodiment, the box illustrated in
In the box structure illustrated in
In the present exemplary embodiment, each box structure is defined as described above, to thereby reduce the amount of data to be described in an image tile. However, any other box structure may be used, as long as a method for grouping and defining metadata (properties) to be applied to two or more items (image items) is employed. Further, a data structure that is handled as an entry in a box may be defined and stored as another box. A box 314 is referred to as a CommonItemPropertyBox and is identified by a cipr identifier. The box 314 is defined by a structure illustrated in
The present exemplary embodiment described above illustrates an example where a box representing a property to be applied in common to all items is defined to thereby improve the search efficiency. However, the present exemplary embodiment is not limited to this example. Information for identifying the application to all items as described above may be stored in each item property box defined in an ItemPropertyContainerBox.
A box 315 is referred to as a CommonItemPropertyGroupBox and is identified by an identifier cipg. The box 315 is defined by a structure illustrated in
The ItemPropertyBox 310 includes a structure illustrated in
Next, a method for storing externally defined metadata Exif data) indicated by the boxes 319, 320, and 321 stored in the media data unit 303 as common metadata will be described with reference to
The Exif data block 401 illustrated in
In contrast, the image capturing date/time varies from image to image. In the example of
However, in the example of
Therefore, the metadata processing unit 110 according to the present exemplary embodiment handles a plurality of pieces of GPS information that falls within a specific range as common metadata. Thus, some types of metadata (e.g., GPS information) can be stored in an area for storing common metadata even when the values do not completely match. The range of common metadata to be handled can be separately designated by the user, or can be appropriately determined according to a specific system setting or the like. In the example of
Further, the metadata processing unit 110 according to the present exemplary embodiment extracts all common data in the Exif data blocks corresponding to each image, but instead may extract common data only for specific Exif data. For example, it may be determined whether only the image capturing date/time is common. Alternatively, for example, it may be determined whether the image capturing date/time item is common only for images with the image capturing date/time falling within a specific range, Thus, it may be determined whether metadata of a specific type is common, and it should be noted that there are a variety of methods for determining the specific type.
As described above, the image data storage device 101 according to the present exemplary embodiment extracts common metadata from the metadata (property information) associated with each of a plurality of images stored in the image file (HEIF file) conforming to the image file format. The image data storage device 101 stores the extracted metadata in the metadata area as common metadata. The image data storage device 101 may also store, in the HEIF file, information indicating whether the common metadata is common to all images, or is common to some of a plurality of images. If the common metadata is common to some of the images, item IDs for some of the images (identification information about the images) are stored. Also, for metadata (e.g., Exif data) based on external standards, the image data storage device 101 extracts common metadata from the metadata corresponding to each image, and holds the extracted metadata as common metadata. The processing load to be imposed when a plurality of images stored in the HEIF file is handled can thereby be reduced. The processing load can be reduced, for example, in the case of performing specific processing only on one or more images associated with specific metadata among the plurality of images stored in the HEIF file. Further, processing of searching for a specific image from among the plurality of images stored in the HEIF file can be performed with a low load. In addition, metadata corresponding to each image is shared and stored, which leads to a reduction in the size of the image file.
In the first exemplary embodiment, an example of extracting common metadata and storing the common metadata in an image file during image generation has been mainly described. In a second exemplary embodiment described below, an example of extracting common metadata from images or metadata already stored in an image file will be described in detail,
Programs to be executed by the CPU 509 include an operating system (OS), a driver, and an application. The CPU 509 controls an overall operation of the image data storage device 501, and performs processing for, for example, changing display of the display unit 505 and sending a connection instruction to the LAN control unit 503, based on an instruction input by the user via the user interface 504.
Programs to be executed by the CPU 509 are stored in the storage unit 507, or are received via the LAN control unit 503 to execute encoding or decoding processing on an input video image or an input image.
In a case of performing playback processing on a video image file or an image file by a user operation, the CPU 509 executes decoding, and the decoded data is played back as a video image or an image by the display unit 505. Programs to be executed by the CPU 509 include a program related to data processing in a format conforming to predetermined file format standards related to video images or images. The data processing can include format processing for storing metadata in an image file, processing for analyzing metadata stored in an image file, and image file editing processing. In a case of handling the HEIF file, the CPU 509 performs processing in a format conforming to HEIF. However, the format is not limited to HEIF. Data processing in a video image file format specified in the MPEG or the like, or data processing in a still image format, such as JPEG, may be performed. The storage unit 507 can be implemented by, for example, a hard disk, a Secure Digital (SD) card, CompactFlash®, a flash memory.
The image recognition unit 506 recognizes, for example, a person, an object, a scene based on an image or video image received from the CPU 509, and sends the result to the CPU 509. The temporary storage unit 508 is a main storage unit for the image data storage device 501, and is mainly used as a temporary data storage area when processing of each functional block is executed. The storage unit 507 is a nonvolatile storage unit that stores software programs to be executed by each functional block, Programs stored in the storage unit 507 are transferred to the temporary storage unit 508 and are read out and executed by each functional block. The LAN control unit 503 is a communication interface for connecting to a LAN, and executes communication control using a wired LAN or wireless LAN. In the present exemplary embodiment, the functional blocks are configured using different circuits, but instead one or more functional blocks can be executed by the CPU 509. The functions illustrated in
Next, a processing flow for editing an image file (HEIF file) conforming to an image file format will be described with reference to
In step S601, the CPU 509 determines the value, range, type, or the like of metadata to be extracted from among metadata associated with each of a plurality of images stored in the image file (HEIF file) conforming to the image file format. This determination can be made based on a user designation operation from the user interface, or can be made based on a predetermined system setting or the like. Moreover, a plurality of determinations can be made.
In step S602, the CPU 509 acquires metadata on each of the plurality of images stored in the HEIF file. The CPU 509 can process not only a single HEIF file, but also a plurality of HEIF files. The CPU 509 can process not only the HEIF file, but also a REG file or the like. The CPU 509 can acquire only metadata already recorded on the HEIF file, or can acquire metadata newly generated through editing processing performed by the image data storage device 501, or through processing performed by the image recognition unit 506.
In step S603, the CPU 509 determines whether the value of the metadata acquired in step S602 matches the value or range of the metadata determined in step S601. If the value of metadata matches the value or the range of the metadata (YES in step S603), the processing proceeds to step S604. If the value of the metadata does not match the value or range of the metadata (NO in step S603), the processing proceeds to step S605.
In step S604, the CPU 509 stores the item ID for identifying the matched image in the storage area for common metadata. In a case of processing a plurality of image files, an item ID for identifying an image item from the other image items is stored in a single HEIF file newly generated.
In step S605, the CPU 509 determines whether the processing on all images from which metadata is to be acquired is completed. If the processing is completed (YES in step S605), the processing proceeds to step S606. If the processing is not completed (NO in step S605), the processing returns to step S602 and the processing is repeated.
In step S606, the CPU 509 deletes metadata that is not common among the other images. In other words, the CPU 509 does not store metadata to be applied only to a single image as common metadata. In step S607, the CPU 509 stores the common metadata in the HEIF file. In this case, based on the common metadata and the image ID, the common metadata is stored in the HEIF file according to the formats illustrated in
As described above, the image data storage device 501 according to the present exemplary embodiment extracts common metadata from the generated image file, and generates a new HEIF file in which the common metadata is stored. With this configuration, an image file in which common metadata is stored can be generated from an image file on which common metadata is not recorded. The image data storage device 501 according to the present exemplary embodiment extracts common metadata from a plurality of images (including a video) stored in a plurality of image files (including a video file), and generates a new image file in which the common metadata is stored.
Thus, it is possible to generate a single HEIF file by editing processing on image files generated by a plurality of devices, or image files generated under different conditions. In this case, as described in the first exemplary embodiment, information indicating whether the common metadata is common to all image groups, or whether the common metadata is common to some of the image groups may be stored. If the common metadata is common to some of the image groups, an item ID for an image corresponding to the common metadata is further stored. Also, for metadata (e.g., Exif data) based on external standards, the image data storage device 501 extracts common metadata from metadata corresponding to each image, and holds the extracted metadata as common metadata. The processing load to be imposed when a plurality of images stored in the HEIF file is handled can thereby be reduced. For example, the processing load can be reduced in the case of performing specific processing only on one or more images associated with specific metadata among a plurality of images stored in the HEIF file. Further, processing of searching for a specific image from among the plurality of images stored in the HEIF file can be performed with a low load. In addition, metadata corresponding to each image is shared and stored, which leads to a reduction in the size of the image file.
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g, application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
Number | Date | Country | Kind |
---|---|---|---|
2018-182086 | Sep 2018 | JP | national |
This application is a Continuation of International Patent Application No. PCT/JP2019/035629, filed Sep. 11, 2019, which claims the benefit of Japanese Patent Application No. 2018-182086 filed Sep. 27, 2018, both of which are hereby incorporated by reference herein in their entirety.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2019/035629 | Sep 2019 | US |
Child | 17210272 | US |