Patent Application
20060115235
This application is based on and claims priority to Japanese patent application JP 2004-293373, filed on Oct. 6, 2004 in the Japanese Patent Office, which is hereby incorporated by reference in its entirety.
1. Field of the Invention
The present invention relates to a technique of recording and reproducing moving pictures.
2. Description of the Related Art
A moving picture recording apparatus performing recording of moving pictures such as a handheld video camera and an installation type monitoring camera has been utilized for various applications. A video picture photographed by a moving picture recording apparatus is recorded on a recording medium such as a video tape, DVD, hard disk (HD) and memory, and is reproduced by a moving picture reproducing apparatus. Also, there is a moving picture recording apparatus of a type having a moving picture reproducing function. In this type of apparatus, a single apparatus can perform recording and reproduction of a moving picture and outputting of a video signal to an external display apparatus.
Most photographing subjects of the moving picture recording apparatus are people and, in many cases, the subjects are human face portions. Consequently, various techniques for improving image quality of a person and a face portion have been conventionally considered.
For example, conventionally, there has been proposed a technique in which an area of a face or a person is detected from an image and the area is subjected to partial high image quality encoding or partial magnification encoding to thereby display a face of a visitor or a prowler clearly on a small screen of a mobile terminal. Furthermore, conventionally, there has been proposed a technique using a so-called ROI (Region of Interest) method; a compression ratio of the area of a person in an image is kept low to thereby keep the image quality of the portion high, while a compression ratio of the other background portion is set high, so that a compression ratio is increased as a whole to thereby reduce the whole capacity of the image after compression processing. Furthermore, conventionally, there has been proposed a technique in which data compression processing is executed for a face area of an image and a background area is replaced by representative compressed data to reduce the capacity of the whole image.
Thus, the prior art attempts simultaneous solution of the contradictory problems of image quality improvement and capacity reduction, in which the quality of face and person portions is maintained, while the quality of the other portions is sacrificed.
A larger capacity recording medium and a higher communication rate have been solving the recording capacity problem associated with recording a moving picture. Accordingly, from now on, demands on image quality such as a demand that the whole appearance of the moving picture including the background is fine, while person and face portions are finer than ever is expected to increase. For example, there is a presumed need that, in situations where a user photographs an athletic meeting of his or her child, the user photographs how his or her child competes with other children, the user photographs scenes of the athletic meeting and the like, and at the same time, clearly records in close-up his or her child's running and facial expressions.
However, in a conventional moving picture recording apparatus, when zooming out for photographing the whole image, a person or a face portion becomes relatively small, so that facial expressions and the like cannot be recorded clearly. On the contrary, when zooming in for photographing a close-up image, the surrounding landscape is cut from an angle of field, so that the whole appearance cannot be obtained. Thus, when photographing his or her child's running race, the user tries to photograph the whole appearance of the running race; resulting in failure in photographing lively expressions of the child, or the user pursues only close-up of the child, resulting in missing a scene in which the child overtakes another child. Therefore, in many cases, the user cannot perform the photographing he or she desires. In particular, when photographing a moving subject, it is difficult to follow the motion of the subject while switching zoom settings or getting a close-up of the face.
The present invention is made in light of the above-described actual situations, and its object is to provide a technique by which whole and close-up high-quality moving pictures can be easily photographed, recorded and reproduced.
In order to achieve the above-described object, in the present invention, photographing, recording or reproduction is performed by the following means or processing.
A moving picture recording apparatus of the present invention comprises imaging means, whole image generating means, partial image generating means and recording means. The imaging means picks up an image at a higher resolution than a resolution of pixels for moving picture recording. Here, the image obtained by picking up the image is referred to as “original image.” The whole image generating means converts the resolution of the original image into an image of the pixels for moving picture recording. The image generated by the whole image generating means is referred to as “whole image.” On the other hand, the partial image generating means generates an image of the pixels for moving picture recording from a part of the original image. The image generated by the partial image generating means is referred to as “partial image.” The recording means records a whole moving picture composed of the whole images and a partial moving picture composed of the partial images in association with each other.
According to this embodiment, only the whole appearance in an angle of field is photographed, yet two types of moving pictures of whole and partial (close-up) moving pictures are automatically recorded; thus, a zooming operation or the like by a photographer is unnecessary, which simplifies photographing and recording. In addition, since either image can be generated from the high-resolution original image, favorable image quality can be obtained.
From which area of the original image the partial image is generated may be fixedly set or may be changed dynamically.
For example, in the case where an area that the user wants to photograph in close-up can be specified in advance, such as a moving picture apparatus for monitoring an entrance of a building and a site (monitoring camera), the area in the original image for generating the partial image may be fixedly set. Furthermore, in the case where there are a plurality of areas that the user wants to photograph in close-up, the areas may be switched at constant time intervals.
Furthermore, by utilizing an image processing technique and an image recognizing technique to detect a subject (person, face or object) to be photographed in close-up from the original image, the area of the partial image can be automatically switched.
Particularly, the partial image generating means may have face detecting means for detecting a human face from the original image and generating the partial image so as to contain the detected face. According to this embodiment, only a person to be a subject in an angle of field is photographed, yet the partial moving picture pursuing the faces of the person in close-up is automatically recorded.
Here, it is preferable that the partial image generating means has face information registering means in which face information is registered and face recognizing means for determining whether or not the detected face coincides with a face registered in the face information registering means, and that in the case of the registered face, the partial image containing the face is generated. According to this embodiment, since the partial moving picture of a non-registrant is not generated, there is an advantage that only the partial moving picture of a desired person (registrant) is recorded even if a number of people are contained in the angle of field.
It is preferable that the partial image generating means extracts an area with a size of the pixels for moving picture recording from the original image and makes it the partial image. In this case, since resolution conversion is unnecessary, simplification and speeding-up of the partial image generation processing can be achieved, and deterioration in image quality due to the resolution conversion can be prevented.
Alternatively, the partial image generating means may extract an area with a size substantially proportional to a size of the face and then convert the resolution of the extracted image into the partial image of the pixels for moving picture recording. In this case, the size of the face occupying the partial image becomes substantially constant. Namely, there is an advantage that the face can be photographed at a constantly similar size.
It is preferable that the partial image generating means extracts the above-described area from the original image so as to keep a position of the face in the partial image substantially constant. Thereby, a partial moving picture with little shaking can be obtained, which is easy to view.
It is preferable that the recording means records the whole moving picture and the partial moving picture while making them temporally correspond to each other. This makes it easy to display the whole moving picture and the partial moving picture in sync when reproducing the recorded moving pictures.
Furthermore, it is preferable that the recording means records a position of the partial image in the whole image. This allows various display effect of associating the whole image and the partial moving picture with each other to be easily realized when reproducing the recorded moving picture.
It is preferable that the recording means has a recording medium removable from a body of the moving picture recording apparatus. This makes it easy to pass the recorded moving picture to other equipment.
It is also preferable that the moving picture recording apparatus comprises image synthesizing means for generating a video signal for displaying the whole moving picture and the partial moving picture while making them correspond to each other. Furthermore, the moving picture recording apparatus may comprise display means for displaying the video signal. Here, the image synthesizing means and the display means each may have a configuration integrated with the body of the moving picture recording apparatus or they may each have a separate configuration.
A moving picture reproducing apparatus of the present invention comprises reading means for reading a whole moving picture and a partial moving picture recorded by the above-described moving picture recording apparatus from the moving picture recording apparatus, image synthesizing means for generating a video signal for displaying the whole moving picture and the partial moving picture while making them correspond to each other, and display means for displaying the video signal.
When the moving picture recording apparatus or the moving picture reproducing apparatus reproduces and displays the moving pictures, for example, the image synthesizing means may synthesize a mark indicating the area of the partial moving picture in the whole moving picture. This allows a user (viewer) to easily grasp what area is being displayed in close-up.
Furthermore, it is preferable that, when there are a plurality of partial moving pictures associated with the whole moving picture, the user is allowed by the operation means to select the partial moving picture to be displayed on a screen. This allows the portion desired by the user to be displayed in close-up. For the modes of a selection screen, there can be various modes such as e.g., a mode in which the partial moving pictures are displayed side by side, a mode in which a plurality of marks indicating the respective areas of the partial moving pictures are synthesized and displayed on the whole moving picture, and a mode in which an arbitrary position in the whole moving picture is selected and a partial image that is the closest to the selected position is displayed.
The present invention includes a moving picture recording apparatus or a moving picture reproducing apparatus having at least a part of the above-described embodiments. Also, the present invention includes a moving picture recording method or a moving picture recording and reproducing method including at least a part of the above-described processing, or a program for realizing the method. Each of the above-described configurations and processing can be combined with each other wherever possible to constitute the present invention.
According to the present invention, a whole moving picture and a close-up moving picture can be easily photographed, recorded and reproduced.
Hereinafter, referring to the drawings, preferred embodiments of this invention are described in detail.
As shown in
The imaging unit 10 (or imaging means) comprising an optical system and an imaging element. The imaging element is an image sensor such as CCD or CMOS imager, for which an element having at least several million effective pixels (for example, 3,000,000 pixels) is used. An image picked up by the imaging unit 10 (referred to as an “original image”) is temporarily memorized in the frame memory 11.
The image processing unit 12 has a function of reading the original image from the frame memory 11 and applying various types of image processing to the original image. Image processing includes for example, resolution conversion, trimming, hand shake correction, sharpness, color tone correction and the like.
Photographing modes of the moving picture recording apparatus 1 include a moving picture photographing mode and a still picture photographing mode; furthermore, the still picture photographing mode includes a high-definition mode and a normal mode. In the high-definition mode, a still picture is recorded at approximately 2000×1500 pixels, which is almost the same as the original image, and in the normal mode, at approximately 1600×1200 pixels. Furthermore, in the moving picture photographing mode, a moving picture is recorded at pixels for moving picture recording of 640×480 pixels (VGA size). In this manner, since the recording pixels are different depending on the photographing modes, the image processing unit 12 applies resolution conversion and trimming to the original image according to the photographing modes as necessary to generate an image for recording.
In the present embodiment, in the moving picture photographing mode, two types of images each having the VGA size (pixels for moving picture recording) of a whole image and a partial image are generated from the original image of approximately 2000×1500 pixels. Whole image generating means comprises the image processing unit 12. Furthermore, partial image generating means comprises the image processing unit 12, the face processing unit 13 and the face information registering unit 14.
In accordance with the invention, the whole image indicates an image containing almost the whole image appearing in the original image. This whole image is generated by converting (down-converting) the resolution of the whole effective area of the original image into the VGA size. The effective area indicates a portion excluding a margin for hand shake correction and excessive pixels for aspect ratio adjustment or the like from the original image and substantially corresponds to the whole area of the original image.
Meanwhile, the partial image indicates an image containing only a part of the image appearing in the original image. This partial image is generated by trimming (extracting) an area of the VGA size from the effective area of the original image. In the present embodiment, the trimming range is determined so as to contain a face portion detected by the face processing unit 13.
Although the whole image and the partial image are generated from the same original image, the whole image is generated from the almost whole area of the original image while the partial image is generated from a smaller area than the whole image. Accordingly, the partial image is such an image that a part of the whole image (e.g., face portion) is in close-up.
The face processing unit 13 has a function of executing face detection processing to detect a human face from the original image (face detecting means) and a function of executing face recognition processing to recognize the detected face (face recognizing means). In the face detection processing, the face processing unit 13 detects an area having facial features from the original image, and calculates coordinate values of a rectangle surrounding the face (referred to as) a “face rectangle”. Furthermore, in the face recognition processing, the face processing unit 13 extracts feature amounts (referred to as face information) of face features such as eyebrows, eyes, nose, mouth and outline from a small image surrounded by the face rectangle, and compares it with face information registered in the face information registering unit 14 in advance to determine whether or not the detected face and the registered face coincide with each other.
Recording means for recording moving pictures comprises the whole image recording unit 15 and the partial image recording unit 16. When sequentially receiving whole images from the image processing unit 12, the whole image recording unit 15 performs moving picture encoding processing to generate a whole moving picture. Furthermore, when sequentially receiving partial images from the image processing unit 12, the partial image recording unit 16 performs moving picture encoding processing to generate a partial moving picture. Then, the whole image recording unit 15 and the partial image recording unit 16 record the whole moving picture and the partial moving picture generated from the same original image on a recording medium in association with each other.
The recording medium is removable from a body of the moving picture recording apparatus 1. As the removable recording medium, for example, a semiconductor memory, CD, DVD, hard disk and the like are assumed.
The image synthesizing unit 17 has a function of reading the whole moving picture and the partial moving picture from the recording medium and performing decoding processing to generate a video signal. This video signal is displayed on the display unit 18 composed of a liquid crystal display, or is output to external equipment such as a printer, display apparatus and recording apparatus (DVD recorder, HD recorder or the like) via an external output I/F.
In configuration example 1, a moving picture recording apparatus comprises a single digital video camera 20 with a moving picture reproducing function that comprises a display unit 21.
In configuration example 2, a moving picture recording apparatus comprises a digital video camera 22 and a television receiver 23. In this configuration, the television receiver 23 takes on the function of the display unit 18 among components of the moving picture recording apparatus, the other functions are provided in the digital video camera 22. A whole moving picture and a partial moving picture photographed and recorded by the digital video camera 22 are output as a video signal by the digital video camera 22 and displayed on the television receiver 23.
In configuration example 3, a moving picture recording apparatus comprises a video camera 24, information processing apparatus (e.g., a personal computer or the like) 25 and a display 26. In this configuration, the video camera 24 takes on the function of the imaging unit 10, and the display 26 takes on the function of the display unit 18, and the information processing apparatus 25 takes on the functions of the other components. The information processing apparatus 25 takes in a video picture from the video camera 24 and generates and records a whole moving picture and a partial moving picture. The recorded moving pictures are reproduced by the information processing apparatus 25 and displayed on the display 26.
In configuration example 4, a moving picture recording apparatus comprises a digital video camera 27, player 28 and a television receiver 29. In this configuration, the player 28 takes on the function of the image synthesizing unit 17, the television receiver 29 takes on the function of the display unit 18, and the digital video camera 27 takes on the functions of the other components. A whole moving picture and a partial moving picture generated by the digital video camera 27 are recorded on a recording medium such as a semiconductor memory 30 and a DVD 31. The player (moving picture reproducing apparatus) 28 reads the moving pictures from the recording medium to generate a video signal. The video signal is displayed on a television receiver 29. Recording the moving pictures on the removable recording medium makes it easy to pass the moving pictures to other equipment.
As described above, the moving picture recording apparatus may have the integrated configuration (configuration example 1), or may have a separate configuration comprising a plurality pieces of equipment (configuration examples 2 to 4). Furthermore, the configuration may be modified or combined as necessary and is not limited to the illustrated examples.
Next, moving picture recording processing of the moving picture recording apparatus is described in detail. FIGS. 3 to 6 are flowcharts showing the moving picture recording processing in the first embodiment of the present invention. The processing described below is realized by a controller (e.g., CPU) of the moving picture recording apparatus executing control of hardware and arithmetic processing according to a program.
When moving picture recording starts, the processing in
The imaging unit 10 first picks up an image (step S10). In the imaging unit 10, the image is picked up at a higher resolution (e.g., 3,000,000 pixels) than the pixels for moving picture recording (VGA size). An original image obtained by picking up the image is stored in the frame memory 11.
Next, the face processing unit 13 creates a partial image list from the original image (step S11). More particularly, as shown in
After the face rectangle is obtained, the face processing unit 13 extracts face information from a face image surrounding by the face rectangle 40 (step S113). The face processing unit 13 checks whether or not the detected face coincides with a registered face, referring to the face information registered in the face information registering unit 14 (step S114). When the detected face coincides with the registered face (step S115; YES), the face processing unit 13 adds a registration ID of the face and the face rectangle to the partial image list (step S116). When there is no coincident face (step S115; NO), the addition to the partial image list is not performed. When the processing for the face rectangle 40 is finished, similar processing is executed for the other face rectangle 41 (step S112).
One example of the partial image list is shown in
The partial image list created in the face processing unit 13 is passed to the image processing unit 12. The image processing unit 12 checks whether the number of items on the partial image list is 0 or more than 0 (step S12). When the number of items is 0, namely when no face is detected, only a whole image is generated and recorded (step S14). Meanwhile, when the number of items is more than 0, namely when at least one face of the registrant is detected, not only the whole image but a partial image of the detected face (close-up image) is generated and recorded (steps S13, S14).
The image processing unit 12 first reads an original image from the frame memory 11 (step S131). Next, the image processing unit 12 acquires coordinates of a face rectangle from the partial image list to calculate coordinates of an area to be extracted as a partial image (step S132). Here, the coordinates are determined so that the center of the face rectangle coincides with the center of the area of the partial image. This is to keep a position of the face in the partial image constant. For example, in an example of
The image processing unit 12 extracts the above-described area from the original image to generate the partial image of the VGA size as shown in
The partial image recording unit 16 first checks whether or not a partial moving picture whose registration ID coincides (limiting to a recorded one) exists. If it exists, the partial image recording unit 16 encodes the received partial image to add to the partial moving picture being recorded. If it does not exist, the partial image recording unit 16 newly generates a partial moving picture with respect to the registration ID. Thus, in the present embodiment, the generation and recording of the partial moving picture are performed for each registrant (for each face) (step S134).
Each frame of the whole moving picture stream 50 is composed by frame data (whole image), a pointer and a time stamp. When a partial image corresponding to the relevant frame exists, information for specifying the partial moving picture stream 51 (for example, a file name or a track on which it is recorded) is stored in the pointer and information for specifying the frame of the partial image (for example, a time, a frame number or the like) is stored in the time stamp. When one frame corresponds to a plurality of partial images, the pointer to each partial image and the time stamp thereof are stored. This allows the whole moving picture and the partial moving picture to be associated with and temporally correspond to each other.
Furthermore, each frame of the partial moving picture streams 51 is composed by frame data (partial image) and position information. The position information stores coordinate values indicating a position (area) of the partial image in the whole image. This allows the whole moving picture and the partial moving picture to positionally correspond to each other.
Next, processing for displaying the whole moving picture and the partial moving picture while make them correspond to each other is described.
The image synthesizing unit 17 reads from a recording medium the whole moving picture and the partial moving picture to be displayed and applies decoding processing, and then applies resolution conversion to the whole moving picture and the partial moving picture as necessary in accordance with their display sizes, respectively. The image synthesizing unit 17 synthesizes the whole moving picture and the partial moving picture to generate a video signal and output the video signal to the display unit 18. This allows the whole moving picture and the partial moving picture to be displayed in sync.
Other display modes include, for example, a mode of displaying the whole moving picture and the partial moving picture in the same size on the screen, a mode of displaying any one of the whole moving picture and the partial moving picture, a mode of synthesizing and displaying two or more partial moving pictures in the whole moving picture, or the like.
The viewer can switch the display mode by operating operation means such as a button and a remote controller of the moving picture recording apparatus. Also, when there are a plurality of partial moving pictures associated with the whole moving picture being displayed, by operating the operation means, the partial moving picture to be displayed on the screen can be switched and the plurality of partial moving pictures can be displayed simultaneously.
Furthermore, in the display examples of
According to the above-described moving picture recording apparatus of the present embodiment, since two types of moving pictures of whole and partial (close-up) moving pictures are automatically recorded merely by photographing the whole appearance in an angle of field, a zooming operation by a photographer or the like is unnecessary, which simplifies photographing and recording. In addition, since either of the images is generated from the original image with a high resolution, favorable image quality can be obtained.
Furthermore, in the present embodiment, since the face to be photographed in close-up is automatically detected from the original image, merely by containing the person to be a subject in the angle of field to photograph, the partial moving picture pursuing the face of the person in close-up is automatically recorded. In addition, since a partial moving picture of a non-registrant is not generated, there is an advantage that, even if there are contained a number of people within the angle of field, only the partial moving picture of a desired person (registrant) is recorded.
Furthermore, in the present embodiment, by trimming the area of the VGA size from the original image, the partial moving image is generated. Therefore, resolution conversion is unnecessary, so that the simplification and speeding-up of the partial image generation processing can be achieved and deterioration in image quality due to the resolution conversion can be prevented.
Furthermore, while, in the conventional moving picture apparatus, the influence of the motion of a subject and hand shaking makes it difficult to follow the motion of a face while obtaining a close-up of the face, in the moving picture recording apparatus of the present embodiment, the trimming of the partial image is performed so as to keep the position of the face in the partial image substantially constant, and thus a moving picture with little shaking can be obtained which is easy to view.
A second embodiment of the present invention is now described. While, in the first embodiment, a partial moving picture is recorded for each registrant by face recognition, in the present embodiment, partial moving pictures are recorded based on the size of the faces in a descending size order. Specifically, creating processing of the partial image list is different.
As in the first embodiment, the face processing unit 13 first reads an original image from the frame memory 11 (step S110). Faces are detected from the original image and face rectangles are calculated (step S111).
After the face rectangles are obtained, the face processing unit 13 sorts the face rectangles in order of reducing the size (step S117). After sorting, the face rectangles other than the top two may be deleted or the face rectangles smaller than predetermined threshold values may be deleted to select only large face rectangles.
The face processing unit 13 adds the face rectangles after sorting to the partial image list in order (steps S118, S119). In the present embodiment, instead of the registration ID, the order after sorting (1, 2, . . . ) is recorded. The processing thereafter is similar to that of the first embodiment.
According to the processing of the present embodiment, a face with a large size is automatically selected as a recording object of the partial moving picture. The photographer normally performs photographing so that a subject that he or she wants to photograph appears large, and thus, the probability that the recording object of the partial moving picture coincides with the subject is high. Namely, in the present embodiment, the partial moving picture of the desired subject can be automatically recorded without performing the registration of the face information and the face recognition processing.
Furthermore, as shown in
A third embodiment of the present invention is now described. While, in the first embodiment, the partial image of the VGA size is trimmed from the original image, in the present embodiment, a partial image is generated so as to keep the size of a face in the partial image substantially constant.
The image processing unit 12 first reads an original image from the frame memory 11 (step S131). Next, the image processing unit 12 acquires coordinates of a face rectangle from the partial image list to calculate coordinates of an area to be extracted as a face image (step S135). The face image indicates an image containing a face (face rectangle) and having a size substantially proportional to the face (face rectangle). In the present embodiment, the area of the face image is determined so as to be about 1.5 to 2 times as large as the face rectangle. An aspect ratio of the face image conforms to the pixels for moving picture recording (VGA size).
The image processing unit 12 extracts the face image from the original image (step S136) and then converts the resolution of the face image into an image of the VGA size to generate the partial image (step S137). The processing after this is similar to that of the first embodiment.
According to the present embodiment, the size of the face occupying the partial image is kept substantially constant. Namely, there is an advantage that the faces can be constantly photographed in the same size.
As described above, while the specific examples of the present invention are shown by the first to third embodiments, the scope of the present invention is not limited to the above-described embodiments, but various modifications can be made within the scope of its technical idea.
While, for example, the cases where the present invention is applied to a handheld video camera are shown in the above-described embodiments, the present invention can be preferably applied to other applications such as a monitoring camera. In the case of an installation type moving picture recording apparatus such as the monitoring camera, an area extracted as a partial image may be fixed.
Furthermore, by combining the processing of the first embodiment and the second embodiment, a partial moving picture may be recorded only in the case where a face has a large size and the face has been registered. Alternatively, the second embodiment and the third embodiment may be combined.
Furthermore, while, in the above-described embodiments, the information associating the whole moving picture and the partial moving picture with each other is recorded in the stream, it is preferable that the whole moving picture and the partial moving picture are associated with each other by utilizing a mechanism such as a multi-angle track of DVD.
Furthermore, in the above-described embodiments, while the comparison between the detected face and the face of the registrant is checked with respect to all frames, for example, the comparison may be performed only one time in 60 frames and in the rest frames, the movement of the face rectangles may be tracked. This can speed up processing.
Furthermore, the stream structures of the moving pictures and the structure of the partial image list described in the above embodiments are only examples and can be changed as necessary. The effective pixels of the imaging unit 10 and the pixels for moving picture recording can also be changed as necessary.
Furthermore, while the face portion is extracted as a partial image in the above-described embodiments, a whole body or an upper body may be extracted as a partial image, for example, by estimating a height of a person from the size of a face or a face rectangle.
| Number | Date | Country | Kind |
|---|---|---|---|
| JP 2004-293373 | Oct 2004 | JP | national |
