Patent Grant
9467621
1. Field of the Invention
The present invention relates to a photographing device, such as a digital camera, a video camera, and a mobile telephone with camera, and in detail, to a photographing device having a function of capturing a still image and motion picture, a photographing method, and a playback method.
2. Description of the Related Art
When photographing is performed, photographing parameters, such as an aperture and zoom position, are switched generally. This method is effective when capturing a still image while switching photographing parameters before photographing in a trial-and-error manner to confirm its effect.
On the other hand, when capturing a motion picture, if photographing parameters are switched during photographing in order to introduce variation in a monotonous screen, camera shake will occur during the operation. Further, the entire process of the trial-and-error operation to switch photographing parameters is photographed. Hence, a general user is reluctant to switch photographing parameters during the photographing of a motion picture. As a result, a monotonous motion picture lacking interest is obtained.
Further, when capturing a motion picture, it is difficult to know when to start and stop photographing, and only a motion picture of an athletic meeting or wedding ceremony, which is a formalized event, is captured with a movie camera. Capturing a motion picture in such a case is likely to result in a monotonous motion picture lacking interest, with a similar composition and exposure.
As described above, when capturing a motion picture, it is likely that only a monotonous motion picture lacking interest is obtained, and there have been made various proposals to solve such a problem. For example, Japanese Unexamined Patent Publication No. 2003-110995 (disclosed on Apr. 11, 2003) discloses an image processing device capable of sequentially connecting a plurality of intervals specified by a user and imparting a transition effect at a connected part. According to this image processing device, it is possible to easily perform a high-level edition of a motion picture desired by a user without the need to be aware of expertise.
Further, Japanese Patent No. 4240108 (disclosed on May 15, 2008) discloses an image sensor capable of detecting a change in facial expression and of photographing at a timing intended by a user. According to this image sensor, it is possible to prevent the timing of photographing from being delayed and to ensure recording of a facial expression desired by a user.
Furthermore, a movie camera capable of capturing a still image while capturing a motion picture is disclosed in the following prior art. Japanese Unexamined Patent Publication No. 4-331583 (disclosed on Nov. 19, 1992) discloses that a strobe zoom mechanism is not used at the time of strobe flash when capturing a still image while capturing a motion picture but an image signal is amplified in an AGC circuit in accordance with information about distance. Further, Japanese Patent No. 3673528 (disclosed on Mar. 10, 1995) discloses a camera capable of easily selecting the priority of still image photographing or the priority of motion picture photographing by providing a motion picture photographing switch and a still image photographing switch.
An object of the present invention is to provide a photographing device capable of capturing and playing back a varied moving image, a photographing method, and a playback method.
The photographing device according to the present invention comprises: an imaging section that captures a subject image and outputs image data; a motion picture recording section that records a motion picture based on the image data; a parameter setting section that sets a photographing parameter to change a photographing state; and a parameter control section that automatically changes the photographing parameter during the recording of the motion picture regardless of the set photographing parameter.
The photographing device according to the present invention comprises: an imaging section that captures a subject image and outputs image data; a motion picture recording section that records a motion picture based on the image data; a parameter setting section that sets a photographing parameter to change a photographing state; a composition change determining section that determines a change in composition in a photographing screen; and a parameter control section that changes the photographing parameter during the recording of the motion picture regardless of the set photographing parameter when the composition change determining section determines that the composition is stable.
The photographing device according to the present invention comprises an imaging section that captures a subject image and outputs image data; a motion picture recording section that records a motion picture based on the image data; a parameter setting section that manually sets a photographing parameter to change a photographing state; a facial expression determining section that determines a change in facial expression of a subject based on the image data; and a parameter control section that changes the photographing parameter during the recording of the motion picture regardless of the manually set photographing parameter in accordance with change in facial expression determined by the determining section.
The photographing method according to the present invention comprises the steps of: capturing a subject image and outputting image data; recording the image data; and changing the photographing parameter during the recording of the motion picture based on the image data regardless of the manually set photographing parameter to change a photographing state.
The photographing device according to the present invention comprises: an imaging section that captures a subject image and outputs image data; a parameter control section that controls a photographing parameter to change a photographing state when acquiring the image data; a release section to instruct to capture a still image; and a photographing control section that acquires the image data in accordance with the instruction to capture a still image and subsequently, while changing the photographing parameter by the parameter control section, acquires and records the image data of continuous shooting.
The photographing device according to the present invention comprises: a screen dividing section that divides a photographing screen into a plurality of regions in accordance with information about focus adjustment; a focus adjustment determining section that assigns a weight determined for each divided region in accordance with a screen feature in the photographing screen divided by the screen dividing section and determines an order of focus adjustment, from which divided region to which divided region the focus should be adjusted, in the screen in accordance with the result of the weighting; and a focus control section that drives a photographic lens in accordance with the order of focus adjustment determined by the focus adjustment determining section.
The playback method according to the present invention comprises the steps of: storing a still image and image data of continuous shooting following the still image; rearranging the stored image data in a direction opposite to the direction of the points of time of continuous shooting; and successively playing back and displaying the still image and the images of the continuous shooting rearranged by an image processing section.
The recording medium that supplies a computer-readable program for causing a computer to execute processing according to the present invention causes a computer to execute processing comprising the steps of: storing a still image and image data of continuous shooting following the still image; rearranging the stored image data in a direction opposite to the direction of points of time of the continuous shooting; and successively playing back and displaying the still image and the images of the continuous shooting rearranged by an image processing section.
Hereinafter, preferred embodiments will be described in accordance with the drawings using a camera to which the present invention has been applied.
The imaging section 2 includes a photographic lens (zoom lens) having a zoom function, an exposure control section, such as a shutter and aperture, an image sensor, a drive and read circuit of the image sensor, etc., and converts a subject image formed by the photographic lens into image data by the image sensor and outputs the data.
The face detecting section 2a detects whether or not a person's face is included in the subject image based on the image data output from the imaging section 2 and processed by an image processing section 5 within the control section 1, and when a face is included, detects the position, size, etc., thereof and outputs the detection result to the control section 1. The detection of a face is performed by extracting the shade of parts, such as an eye, nose, and mouth, in a face and using a pattern matching method etc.
The control section 1 controls the total sequence of the camera 10 in accordance with programs stored in the flash ROM 12. The control section 1 has a facial expression change extracting section 1a, a composition change determining section 1b, and the image processing section 5. The facial expression change extracting section 1a extracts a change in facial expression based on the image data output from the imaging section 2, the detection result of a face output from the face detecting section 2a, and the pattern of parts of a face, such as an eye, nose, and mouth, and data about the change in a facial expression stored in a facial expression change database (DB) 6a, to be described later.
The composition change determining section 1b determines whether or not the composition of the screen and the way to hold the camera 10 change. When determining a change in the composition of the screen, the composition change determining section 1b makes use of a motion vector etc. that is obtained based on the image data output from the imaging section 2. The way to hold the camera 10 is determined based on a determination output of the motion determining section 4, to be described later.
The image processing section 5 has a motion picture section 5h and a still image section 5i, takes in image data output from the imaging section 2, performs various kinds of image processing, such as thinning processing, trimming processing, edge enhancement, color correction, and image compression, and performs image processing for displaying a live view of a motion picture, for recording a still image and a motion picture in the recording section 6, for playing back and displaying a still image and a motion picture, etc. The image processing section 5 has a circuit for still image and a circuit for motion picture in accordance with processing, such as a still image compression/decompression circuit, a motion picture compression/decompression circuit, etc., for compressing/decompressing images. Further, the image processing section 5 extracts a high-frequency component of image data and determines a contrast of a subject. Information about this contrast is used for adjusting focus of a photographic lens and also used for making a determination at the time of automatic parameter control, to be described later.
The parameter setting section 3 manually sets photographing parameters and outputs the parameters to the control section 1 using a zoom button to change the focal length in the imaging section 2, an aperture value setting operation section for setting an aperture value, etc. It may also be possible to automatically set an aperture value etc. in accordance with the result of exposure control.
The parameter control section 7 controls the imaging section 2 based on the photographing parameters manually set or automatically set in the parameter setting section 3. By controlling the photographing parameters, it becomes possible to adjust the angle of view, blurred background (depth of field), etc. By changing these parameters, it is possible to apply various effects to enjoy photographing. Further, the parameter control section 7 detects the currently set values, such as the aperture value and focal length, and output the values to the control section 1.
The motion determining section 4 has a hexaxial sensor etc. that detects physical quantities in six axes in total: an acceleration in the three-dimensional space, that is, a triaxial acceleration; and a rotation speed in the three-dimensional space, that is, a triaxial angular speed. It is also possible for the motion determining section 4 to determine whether or not the way to hold the camera 10 has changed. It is possible to determine that the way to hold the camera 10 of a photographer has not changed even when the image changes due to zooming etc., and it can be known whether or not the photographer aims at the same subject without changing the way to hold the camera without erroneous operations.
The operation section 9 includes various operation members, such as a still image release button, a motion picture button, a power switch, a playback mode setting button, a menu button, and an automatic mode setting button, and determines the operation state of these operation members and sends the determination result to the control section 1. The control section 1 described above controls photographing and playback in a predetermined sequence in accordance with the operation state of the operation member.
Pressing the still image release button instructs to capture a still image and the motion picture button instructs to start capturing of a motion picture when pressed for the first time and instructs to end the capturing of a motion picture when pressed again. The automatic mode setting button is an operation member to set the automatic mode in which photographing parameters are switched automatically in accordance with the facial expression of a subject, as will be described later. When a touch panel is provided on the front surface of the display section 8, it may also be possible to set the automatic mode with the touch panel.
When instructed to photograph with the release button, the recording section 6 records the image data of a still image and a motion picture captured by the imaging section 2 and subjected to compression processing by the image processing section 3. The facial expression change database (DB) 6a within the recording section 6 is a database showing a relationship between the shapes, positions, etc., of the parts of a face, such as an eye, nose, and mouth, and the facial expression. Based on the relationship, it is possible for the facial expression change extracting section 1a to represent the facial expression of a person, such as the degree of smile, by a numerical value. The degree of smile is represented by a numerical value based on the shape of the eye and mouth, whether or not teeth are seen, etc. The flash ROM 12 is an electrically rewritable memory and stores control programs to be executed in the control section 1, and data such as adjustment values. A memory other than the flash ROM 12, for example, a mask ROM may be accepted as long as it can store the control programs etc.
The display section 8 has a display section, such as a liquid crystal panel and an organic EL panel, disposed on the backside etc. of the camera 10 and performs a live view display before photographing, normal play back display of a recorded image, display of camera information, etc. It is possible for a photographer to observe the live view display to determine a composition and timing.
Next, the automatic change (automatic mode) of the photographing parameters during the display of a live view and photographing of a motion picture in the present embodiment will be described using
In
In this manner, in the examples shown in
In
In this manner, in the examples in
By performing the automatic parameter control in this manner, in the examples shown in
When the angle of view changes due to zooming, the composition change section 1b detects change in composition. When the change in composition is detected, if the motion determining section 4 determines that the way to hold the camera 10 does not change and the direction of the same person 21 is photographed, it is recommended to perform the automatic parameter control on the assumption that the composition is stable.
Next, the operation of the camera 10 in the present embodiment will be described using the flowchart of camera control shown in
When the mode is the photographing mode according to the result of the determination in step S1, next, zero is set to an automatic flag if it is a timing of the initial setting of the mode setting (S2). When the automatic flag is zero, the mode is the automatic mode and when the automatic mode is set, the photographing parameters, such as the aperture and the focal length, are controlled to switch in accordance with the facial expression of the person 21 during the display of a live view and photographing of a motion picture.
When zero is set to the automatic flag, next, an image is captured and a display is produced (S3). In this step, the image data captured by the imaging section 2 is subjected to image processing in the image processing section 5 and a display of a live view is produced on the display section 8 before capturing a moving image or a still image, and during the photographing of a motion picture, a moving image during the photographing as shown in
Subsequently, the face and the facial expression are detected (S4). In this step, the face detecting section 2a detects whether the person 21 is included in the image data, and when included, the facial expression change extracting section 1a detects the facial expression of the person 21.
When the face and the facial expression are detected, next, whether or not the parameter manipulation is performed is determined (S5). Here, the parameter setting section 3 determines whether or not the manual setting of the aperture value, zooming, etc., is performed. When the parameter manipulation is not performed according to the result of the determination, next, whether or not the automatic setting is performed is determined (S11).
In step 11, whether or not the automatic mode setting button within the operation section 9 is operated, or whether or not the composition of the screen changes for a predetermined period of time is determined. Here, the predetermined period of time is set, for example, to one to three seconds, in which a viewer begins to feel that the composition does not change and is monotonous when viewing a motion picture. In the present embodiment, whether or not the automatic setting is performed is determined based on the operation state of the automatic mode setting button and the change in the composition of the screen, however, this is not limited, and either one will do.
When the automatic setting is performed according to the result of the determination in step S11, zero is set to the automatic flag (S12). As described above, at the time of initialization, zero is set to the automatic flag in step S2, however, when it is determined that the parameter manipulation is performed in step S5, one is set to the automatic flag in step S7, to be described later. When one is set to the automatic flag, zero is set to the automatic flag in step S12.
When the automatic setting is not performed according to the result of the determination in step S11, or when zero is set to the automatic flag in S12, next, whether or not the automatic flag is zero is determined (S13).
When the automatic flag is zero according to the result of the determination in step S13, the parameter automatic control is performed (S14). In the parameter automatic control, the switch control of the photographing parameters, such as the aperture and the focal length, is performed in accordance with the facial expression of the subject person 21, as described in
When the parameter manipulation is performed according to the result of the determination in step S5, the parameters are changed in accordance with the manipulation (S6). In step S5, the manual setting of the aperture value, zooming etc., is performed, and therefore, in step S6, the aperture value, the focal length, etc., are changed in accordance with the manual setting. Subsequently, one is set to the automatic flag (S7). When one is set to the automatic flag, the camera is controlled based on the photographing parameters manually set by a photographer.
When the automatic flag is set to one, next, whether or not the photographing is still image photographing is determined (S15). Here, whether or not the still image release button is pressed and operated is determined. When the photographing is still image photographing according to the result of the determination, still image photographing is performed (S16). In this step, the image data output from the imaging section 2 is subjected to image processing and the image data of the still image is recorded in the recording section 6. When still image photographing is performed, the process returns to step S1.
When the photographing is not still image photographing according to the result of the determination in step S15, next, whether or not the photographing is motion picture photographing is determined (S17). Here, the motion picture button is operated in the motion picture not-photographing state is determined. When the motion picture button is operated, a flag, not shown schematically, is set until the motion picture button is operated again, and the determination result in step S17 is Yes.
When the photographing is motion picture photographing according to the result of the determination in step S17, the motion picture is recorded (S18). Here, the image data output from the imaging section 2 is subjected to image processing and the image data of the motion picture is recorded in the recording section 6. During the motion picture photographing, processing is repeated in the order of step S18→S1→S2→ . . . →S5→S11→ . . . →S15→S17→S18. Hence, when the automatic flag is set to zero, the parameter automatic control is performed in step S14, the photographing parameters are changed in accordance with the facial expression of the subject person 21, and the moving image at this time is recorded in the recording section 6. When the photographing is not motion picture photographing according to the result of the determination in step S17, the process returns to step S1.
When the mode is not the photographing mode according to the result of the determination in step S1, whether or not the photographing mode is terminated during the photographing is determined (S21). When the mode is not the photographing mode, the process advances to step S21, and in addition to this, when the motion picture button is pressed and operated again during the motion picture photographing, the process also advance to step S21. When the motion picture button is operated again during the motion picture photographing, it is determined that the photographing mode is terminated during the photographing. When the photographing mode is terminated according to the result of the determination, the photographing is terminated (S22). Here, motion picture photographing termination processing is performed.
When the termination processing in step S22 is performed, or when the photographing mode is not terminated during the photographing according to the result of the determination in step S21, next, whether or not the mode is the playback mode is determined (S23). Here, whether or not the playback mode setting button of the operation section 9 is operated is determined. When the mode is not the playback mode according to the result of the determination, the process returns to step S1.
On the other hand, when the mode is the playback mode according to the result of the determination in step S23, the playback mode is performed (S24). In the playback mode, a list of the captured images recorded in the recording section 6 is displayed on the display section 8 and an enlarged display of a thumbnail selected by a photographer is produced. When the playback mode is performed, the process returns to step S1.
As described above, in the flowchart of camera control in the present embodiment, when the automatic mode is set, the photographing parameters, such as the aperture and the focal length, are automatically switched in accordance with the facial expression of the subject. Hence, when a live view display is produced or a motion picture is recorded, the image changes in accordance with the facial expression and in particular, when a motion picture is recorded, it is possible to capture a varied image.
The parameter automatic control is performed also during the display of a live view, not limited to during the recording of a motion picture. That is, before capturing a motion picture starts, steps are performed in the order of step S1→ . . . →S5→S11→ . . . S13→S14→S15 NO→S17 NO→S1 and in step S14, the parameter automatic control is performed and the photographing parameters are changed in accordance with the facial expression of the person 21.
Next, the details of the parameter automatic control in step S14 are described using
When a face is detected according to the result of the determination in step S31, next, whether or not the position and direction of the camera are the same for a predetermined period of time is determined (S32). Here, based on the determination results of the motion determining section 4 and the composition change determining section 5, whether or not the camera 10 is photographing the same subject in the same direction is determined. Here, the predetermined period of time is assumed to be, for example, about one to three seconds, in which a viewer begins to feel that the composition does not change and is monotonous when viewing a motion picture.
When the position and direction of the camera are the same for a predetermined period of time according to the result of the determination in step S32, next, whether or not the motion of the background is small is determined (S33). Here, the composition change determining section 5 analyzes the image of the image data and makes the determination based on the motion of the background. This is because when the background is in motion, the image is not monotonous and it is inappropriate to change the photographing parameters.
When the motion of the background is small according to the result of the determination in step S33, next, whether or not the contrast of the background is high is determined (S34). Here, the image processing section 5 extracts a high-frequency component of the image data corresponding to the background part and determines whether or not the contrast is high based on the high-frequency component. When the contrast of the background is higher than the contrast of the face part of the person 21, the background gives an impression of mess, and therefore, the effect to change the aperture is significant.
When the background contrast is high according to the result of the determination in step S34, next, whether or not there is a room to change the aperture is determined (S35). When the subject is dark, if the aperture is reduced, the image becomes dark and conversely, when the subject is bright, it may be overexposed, and therefore, in this step, whether or not the exposure is allowable even if the aperture is changed is determined. This determination is made by acquiring information about the aperture value from the parameter control section 7.
When there is a room to change the aperture according to the result of the determination in step S35, next, the aperture is changed in accordance with the facial expression (S36). In this step, as described using
When the background contrast is low according to the result of the determination in step S34, or when there is not a room to change the aperture according to the result of the determination in step S35, it is not possible to switch the aperture to another, and therefore, the photographing parameter is changed by zooming in step S41 and subsequent steps.
First, whether or not the background pattern position changes is determined (S41). For the composition in which the background is plain or the contrast thereof is not low and the pattern varies from the subject in the center of the screen toward the periphery, it is effective to change the background by zooming. When the background is plain, such as that in a certificate photograph, the change in the screen by zooming is monotonous and not interesting. Hence, in this step, the background patter is determined and whether or not the pattern changes depending on the position in the screen is determined. Only when there is a change, zooming is performed.
When there is a change in the position according to the result of the determination in step S41, next, whether or not the zoom can be changed is determined (S42). Here, whether or not the shortest focal length end or the longest focal length end of the zoom lens is reached is determined.
When the zoom can be changed according to the result of the determination in step S42, the zoom is changed in accordance with the facial expression (S43). In this step, as described using
When a face is not detected according to the result of the determination in step S31, the object is not a person because no face is detected. In this case, the object of photographing can be thought, for example, an accessory, a small item, a dish, etc., and therefore, in step S51 and subsequent steps (refer to
When a face is not detected, next, whether the position remains the same for a predetermined period of time is determined (S51). Here, the composition change extracting section 1b analyzes the image data and determines whether the position of the subject can be changed. Here, the predetermined period of time is set to, for example, about one to three seconds, in which a viewer begins to feel that the composition does not change and is monotonous when viewing a motion picture.
When the position remains the same for a predetermined period of time according to the result of the determination in step S51, next, whether or not the motion of the background is small is determined (S52), similarly as in step S33. This is because performing the parameter automatic control is inappropriate when the background is in motion. The motion of the background is determined in the composition change determining section 1b.
When the motion of the background is small according to the result of the determination in step S52, next, whether or not the background contrast is high is determined (S53), similarly as in step S34. When the background contrast is high according to the result of the determination, next, whether or not there is a room to change the aperture is determined (S54), similarly as in step S35. When there is a room to change the aperture according to the result of the determination, next, whether or not the room is toward the open side is determined (S55).
When there is a room toward the open side according to the result of the determination in step S55, next, the aperture is controlled toward the open side (S56). Here, the parameter control section 7 gradually opens the aperture in the imaging section 2 toward the open side. On the other hand, when there is not a room toward the open side according to the result of the determination in step S55, the aperture is controlled toward the closed side (S57). Here, the parameter control section 7 gradually reduces the aperture toward the fully-closed side.
When the background contrast is not high according to the result of the determination in step S53, the change is small even if the aperture value is changed because the background contrast is low, and therefore, a varied image is not obtained. Hence, the angle of view is changed in step S61 and subsequent steps. First, whether or not the background pattern position changes is determined (S61), similarly as in step S41.
When the background pattern position changes according to the result of the determination in step S61, next, whether or not the zoom can be changed toward the wide side is determined (S62). When there is a room to change the zoom toward the wide side according to the result of the determination, next, zooming is performed toward the wide side (S63). Here, the parameter control section 7 gradually moves the focal length of the zoom lens toward the wide side.
On the other hand, when there is not a room to change the zoom toward the wide side according to the result of the determination in step S62, next, zooming is performed toward the telephoto side (S64). Here, the parameter control section 7 gradually moves the focal length of the zoom lens toward the telephoto side.
When zooming is performed toward the telephoto side in step S64, or when zooming is performed toward the wide side in step S63, or when the aperture is controlled toward the closed side in step S57, or when the aperture is controlled toward the open side in step S56, or when the result of the determination in step S51 or S52 is No, the change of parameters is terminated (S65) if there is an operation or motion, and the process returns to the original flow. When the photographing parameters are changed in accordance with the facial expression, the facial expression changes with time and it is possible to change the photographing parameters in accordance with the change in the facial expression. However, for other than a face, the change of parameters is carried out limitlessly, and therefore, when the composition of a subject is changed, the subject moves, or a photographer performs an operation, the change of parameters is terminated.
As described above, in the first embodiment of the present invention, a motion picture is prevented from becoming monotonous by changing the motion picture, such as blurring the background and zooming in the subject, and thus, the change in the image is made dramatic. If a motion picture is given an effect by gradually blurring the background or zooming in the subject, an image pleasant to view is obtained, however, when a user is absorbed in photographing, it is not possible for the user to perform such an operation quickly. If a user tries to perform the operation forcedly, camera shake will occur. Hence, in the present embodiment, such a kind of transition effect is automatically performed by the camera 10. However, if this is done automatically at all times, an unintended image will be obtained, and therefore, the camera 10 is so designed that such an operation is performed only when predetermined conditions to cause such a transition effect are satisfied.
The predetermined conditions to cause the transition effect include a case where there is expected an effect when the aperture is opened, for example, when there is a room for the position of the aperture and the distance to the subject is small. In this case, it may also be possible to add an image of the background. Another case is also included, where there is a room to change the zoom toward the wide side. It may also be possible to change the zoom toward the telephoto side, however, in this case, the same effect may be obtained by trimming the image after photographing. Further, another case is also included, where there is a change in the facial expression of the subject. In this case, as described in the present embodiment, it may also be possible to change the effect of the background in accordance with the change in the facial expression.
Next, a modified example of the first embodiment of the present invention will be described using
Blurring the background should not be performed simply because the aperture can be changed, but it should be determined whether the scene is such one in which the background includes various objects as shown in
When a photographer permits to blur the background, the parameter control section 7 moves the aperture toward the open side to blur the background and at this time, the control state of the photographing parameter is displayed on the advice display section 8b as “Background is blurred” (refer to
The configuration of the modified example of the first embodiment is the same as that in
Next, a second embodiment of the present invention will be described using
The flowchart of motion picture photographing of a person that enables such moving image photographing is described using
Subsequently, whether or not there is a change in the way to hold the camera is determined (S71). Here, the composition determining section 1b determines whether or not there is a change in the way to hold the camera 10 based on the output of the hexaxial sensor etc. within the motion determining section 4. The motion determination may be made not only by the motion determining section 4 but also by the image processing section 5 based on the change in image. When the way to hold the camera does not change according to the result of the determination, next, whether or not zooming is on the way is determined (S72). There is a case where zooming is performed in step S95 (refer to
When zooming is not on the way according to the result of the determination in step S72, that is, when zoom operation is not being performed, next, whether or not there is a change in composition is determined (S73). Whether or not there is a change in composition is determined by the composition change determining section 1b based on the image data.
When zooming is on the way according to the result of the determination in step S72, next, whether or not there is only a change in image by zooming is determined (S74). Here, whether or not the change in composition is only the change in image by the change in zoom is determined. Specifically, the composition change determining section 1b acquires information about the focal length of the zoom lens from the parameter control section 7, performs image correction of the image data based on the information about focal length, and determines whether or not there is a change in the corrected image.
When there is only a change in image by zooming according to the result of the determination in step S74, or when there is not a change in composition according to the result of the determination in step S73, next, the facial expression change extracting section 1a determines whether or not there is a change in facial expression (S75). When there is a change in facial expression according to the result of the determination, photographing with a transition effect is performed (S78). Here, as described in
When the photographing with a transition effect is performed, next, whether or not the photographing ends is determined (S79). The photographing ends when the motion picture button of the operation section 9 is operated again, and therefore, in this step, the operation state of the motion picture button is determined.
When the photographing does not end according to the result of the determination in step S79, or when there is not a change in facial expression according to the result of the determination in step S75, or when there is not a change in composition according to the result of the determination in step S73, or when there is not only a change in image by zooming according the result of the determination in step S74, next, normal motion picture photographing is performed (S76). Here, the image data output from the imaging section 2 is subjected to image processing and then recorded in the recording section 6. It is of course possible to record as a still image by pressing to operate the still image release button at the timing after the normal motion picture photographing in step S76. In this case, an image, in which the effect of the change in the photographing parameters is reflected, is displayed on the display section 8 and if there is an image that a photographer is interested in, it is possible to perform photographing of a still image at this timing.
Subsequently, whether or not the motion picture photographing is ended is determined (S77). Here, similarly as in step S79, the determination is made based on the operation state of the motion picture button of the operation section 9. When the photographing is not ended according to the result of the determination, the process returns to step S70. Consequently, when the flow of the motion picture photographing of a person is started, normal motion picture photographing is performed continuously in step S76, however, when a monotonous image results even when there is a change in facial expression of a person, that is, the way to hold the camera remains unchanged, the composition is not changed, but there is a change in facial expression of a person, it is designed so that the photographing with a transition effect is performed in step S78.
When the photographing is ended according to the result of the determination in step S77 or S79, next, whether or not the best facial expression of a transition image is included is determined (S81). In step S70, the change in facial expression, such as the degree of smile, is recorded and the photographing with a transition effect in step S78 is performed, and therefore, in this step S81, whether or not the image of the best facial expression is included in the image captured with a transition effect is determined.
When the best facial expression is included in the transition image according to the result of the determination in step S81, next, images other than between the start of the transition to the end thereof are deleted (S82). That is, the normal motion picture photographing is performed, however, if the best facial expression is included in the image captured with the transition effect when the facial expression changes, only the moving images in the interval during which the photographing with the transition effect is performed are left and moving images in other intervals are deleted.
Next, the photographing with a transition effect in step S78 will be described using the flowchart shown in
When the display of instructing to fix the camera is produced, next, whether or not the aperture can be opened fully is determined (S93). Here, information about the current aperture value is acquired from the parameter control section 7 and compared with the fully open aperture value, and then, whether or not the aperture can be still opened toward the fully open direction is determined. When the aperture can be fully opened according to the result of the determination, photographing with fully open aperture is performed (S94). Here, the parameter control section 7 is instructed to drive the aperture fully open.
When the aperture cannot be fully opened according to the result of the determination in step S93, next, photographing toward the telephoto direction is performed (S95). Here, the parameter control section 7 is instructed to drive the zoom lens toward the telephoto side. As described above, when the aperture can be driven fully open, the aperture is fully opened and when the aperture cannot be driven fully open, the zoom lens is driven toward the telephoto side.
When the photographing in the direction of fully open aperture in step S94 is performed, or when the photographing in the telephoto direction in step S95 is performed, next, whether or not transition is completed is determined (S96). Here, whether the fully open aperture value is reached in the fully open aperture photographing or whether the telephoto end is reached in the photographing in the telephoto direction is determined. When the transition is not completed according to the result of the determination, the process returns to step S92.
On the other hand, when the transition is completed according to the result of the determination in step S86, the completion point of time is recorded (S97). The starting point of time is recorded in step S91 and the completion point of time is recorded in step S97, and therefore, the interval during which the transition photographing is performed is recorded.
When the completion point of time is recorded, next, the photographing parameters are reset to the initial ones and photographing is continued (S96) and the process returns to the original flow. In steps S94 and S95, the aperture is driven to the fully open aperture value or the zoom lens is driven to the telephoto end, and therefore, in this step S96, the photographing parameters are reset to those before the drive and the photographing is continued.
As described above, in the present embodiment, when the camera 10 is photographing the motion picture of the same person, in order to prevent an image from becoming monotonous, the photographing parameters are changed and the photographing with a transition effect is performed when the facial expression of the person changes. Further when the best facial expression is included in the images captured by the photographing with a transition effect, the images in the interval are saved and the other images are deleted.
During the normal motion picture photographing, when the photographing with a transition effect is performed a plurality of times, the images in the interval in which the facial expression is best are saved, however, this is not limited, and all of the images captured by the photographing with a transition effect may be saved or the best two more images with the transition effect may be saved.
Next, a third embodiment of the present invention will be described using
In general, when photographing a still image, in most cases, the best facial expression of a subject is captured, and it may also be possible to perform a motion picture photographing of the change in facial expression when the subject relaxes after a still image photographing with the transition effect of an image. In this case, it is convenient if a motion picture is captured automatically with the same operation as that to photograph a still image without the need to set the motion picture photographing purposely. Further, if a motion picture with the transition effect is captured following a still image, it is possible to create a new motion picture by synthesizing the motion picture with another and the synthesized motion picture can be made use of as a raw material. It becomes also possible to represent the passage before a still image is captured by rewinding and playing back the motion picture.
The configuration in the present embodiment is the same as the block diagram shown in
When the still image release button is operated after letting time elapse while waiting, the still image photographing is performed (S102). Here, the image data output from the imaging section 2 is subjected to image processing by the image processing section 5 and the image data of the still image subjected to the processing is recorded in the recording section 6.
When the still image photographing is performed, next, whether or not only one person is in the center is determined (S103) according to the result of face detection by the face detecting section 2a. When there are two or more persons according to the result of the determination, this is not suitable for the photographing while changing the photographing parameters, such as the angle of view and aperture, and therefore, this flow is terminated.
On the other hand, when there is only one person according to the result of the determination in step S103, next, whether or not the face is in a close-up state is determined (S104). Here, whether not the size of the face detected by the face detecting section 2a is in a close-up state where it occupies almost all the area of the screen is determined.
When the face is not in a close-up state according to the result of the determination in this step S104, next, motion picture photographing is performed in a close-up with an overexposure (S121). Here, the motion picture photographing is performed by instructing the parameter control section 7 to gradually drive the focal length toward the telephoto side and at the same time, to gradually change the electronic shutter speed, the ISO sensitivity, etc., so that an overexposure is achieved. When the focal length is near the telephoto end, it may also be possible to achieve a close-up by electronic zooming. Further, the photographing time of the motion picture may be a predetermined period of time or the time until the telephoto end is reached.
In step S121, achieving an overexposure may be omitted, or it may also be possible to perform photographing while changing other photographing parameters instead of achieving an overexposure, for example, while changing the focus position of the zoom lens. Some persons do not desire their faces to be photographed clearly, and in such a case, a close-up is omitted and the exposure and the focus may be changed considerably so that their faces are unclear.
When the face is in a close-up state according to the result of the determination in step S104, next, whether or not the focal length can be shifted toward the wide angle side is determined (S105). Here, the determination is made by acquiring information about the current focal length from the parameter control section 7 and comparing it with the focal length of the wide end.
When the focal length can be shifted toward the wide angle side according to the result of the determination in step S105, next, motion picture photographing is performed while shifting the focal length toward the wide angle side (S106). Here, a motion picture is recorded based on the image data output from the imaging section 2 while instructing the parameter control section 7 to gradually change the focal length toward the wide angle side. The photographing time at this time may be a predetermined period of time, or the time until the wide end is reached.
When the focal length cannot be shifted toward the wide angle side according to the result of the determination in step S105, next, the contrast distribution is determined (S111). Here, the image processing section 5 extracts a high-frequency component of the image data and determines the contrast distribution.
When the contrast distribution determination is made, next, whether or not the effect of aperture can be achieved is determined (S113). When the current aperture value is near the fully open state, the depth of field can be made deeper by reducing the aperture and when the current aperture value is near the smallest aperture, the depth of field can be made shallower by opening the aperture toward the fully open side. In this step, whether or not the effect of aperture can be achieved is determined from the relationship between the current aperture value and the contrast distribution.
When the effect of aperture can be achieved according to the result of the determination in step S112, motion picture photographing is performed while changing the aperture (S113). Here, the parameter control section 7 is instructed to change the aperture value toward the direction in which the effect of aperture can be achieved determined in step S112. The photographing time at this time may be a predetermined period of time or may be the time until the fully open aperture value or the smallest aperture value is reached. On the other hand, when the effect of aperture cannot be achieved according to the result of the determination, the process advances to the above-described step S121 and photographing is performed in a close-up with an overexposure.
When motion picture photographing is performed while changing the aperture in step S113, or when motion picture photographing is performed while shifting the focal length toward the wide angle side in step S106, or when motion picture photographing is performed in a close-up with an overexposure in step S121, next, the still image and the motion picture are associated with each other and then recorded (S122). Here, the image data of the still image captured in step S102 and the image data of the motion picture captured in step S106, S113, or S121 are associated with each other. After the recording is completed, the flow of photographing a person ends.
As described above, in the third embodiment, during the still image photographing, a motion picture is captured while changing the photographing parameters. Hence, it becomes possible to view not only a mere still image but also a varied motion picture. Further, the still image and motion picture can be used as raw materials when editing these images to create a motion picture.
As described above, in each embodiment of the first to third inventions, during the recording of a motion picture based on image data, the photographing parameters are changed in accordance with the motion of a photographer and a subject regardless of the photographing parameters manually or automatically set, and therefore, it is possible to capture a varied image in photographing.
Further, in the embodiments of the present invention, it is made possible to capture a motion picture with changed photographing parameters regardless of the photographing parameters manually set following the recording of a still image, and therefore, it is possible to capture a varied image.
Furthermore, in the embodiments of the present invention, before recording a motion picture and when a live view is displayed, the photographing parameters are changed regardless of the photographing parameters manually or automatically set, and therefore, it is possible to confirm in advance whether or not a varied image can be captured by photographing. When a live view is displayed, the aperture and the angle of view are changed, and therefore, it is possible to capture a still image when an image a photographer is interested in appears.
In the first to third embodiments of the present invention, as the photographing parameters, the aperture value and the focal length (zoom) are described, however, these are not limited, and it may also be possible to sequentially switch the focus positions or change the exposure (exposure correction value).
In the first to third embodiments of the present invention, the degree of blurring as a photographing parameter is changed by controlling the aperture value, however, it may also be possible to change the degree of blurring by performing image processing instead of the aperture when the background is blurred by image processing.
The facial expression change DB 6a in the first to third embodiments of the present invention stores a database indicative of the relationship between the shapes, positions, etc., of the parts of a face, such as eyes, nose, mouth, etc., and the facial expression of the face, however, it may also be possible to store data of speed when the facial expression changes, in addition to the relationship. Further, in the embodiments, the photographing parameters are switched based on the facial expression, such as the degree of smile, however, it may also be possible to control the change of photographing parameters using the data about the speed of change in facial expression while predicting the change. Furthermore, in the embodiments, the facial expression is determined by the face detecting section 2a as a motion of a subject and the photographing parameters are changed based thereon, however, this is not limited and it may also be possible to change the photographing parameters based on the motion of arm, motion of head, etc. Still furthermore, in the embodiments, the motion of a photographer is determined by the motion determining section 4 and whether or not the photographing parameters are changed is determined based on the result of the determination, however, it may also be possible to change the photographing parameters directly in accordance with the motion of a photographer.
Next, a fourth embodiment of the present invention will be described using
Like the imaging section 2 in the first embodiment, the imaging section 2 includes a photographic lens (zoom lens) having a zoom function, exposure control sections, such as a shutter and aperture, an image sensor, drive and read circuits of the image sensor, etc., and converts a subject image formed by the photographic lens into image data by the image sensor and outputs the image data. The photographic lens of the imaging section 2 includes a focus adjustment lens and its mechanism and in the fourth embodiment, it is also possible to capture a transition motion picture while shifting the focus. It may also be possible to provide an aperture adjusting mechanism for blurring the background. The face detecting section 2a is the same as the face detecting section 2a in the first embodiment, and therefore, its detailed description is omitted.
The control section 1 controls the total sequence of the camera 10 in accordance with the programs stored in the flash ROM 12. The control section 1 has the composition change determining section 1b, a trimming section 1c, and the image processing section 5. The composition change determining section 1b is the same as that in
The trimming section 1c performs trimming processing to trim part of image data output from the imaging section 2. The trimming section 1c trims an image in the center of the screen when trimming image data, however, it may also be possible to design the trimming section 1c so as to trim the periphery of the face of a subject based on the position, size, etc., of the face detected by the face detecting section 2a, in addition to the central image.
The image processing section 5 has a temporary recording section 5a, the motion picture section 5h, and the still image section 5i and takes in image data output from the imaging section 2, performs various kinds of image processing, such as thinning processing, trimming processing, edge enhancement, color correction, and image compression, and performs image processing for displaying a live view of a motion picture, recording a still image and motion picture in the recording section 6, and playing back and displaying a still image and motion picture. For image compression/decompression, the image processing section 5 has circuits for a still image and motion picture in accordance with processing, such as a still image compression/decompression circuit and a motion picture compression/decompression circuit. It is also possible for the image processing section 5 to acquire a motion vector based on image data. As will be described later, the temporary recording section 5a is a memory for temporarily recording image data of a motion picture captured following the still image photographing. Details of the image processing section 5 will be described later using
The operation section 9 includes various operation members, such as a release button, a power switch, a playback mode setting button, a menu button, and an art mode button, and determines the operation state of these operation members and sends the determination result to the control section 1. The control section 1 described above controls photographing and playback in a predetermined sequence in accordance with the operation state of the operation members. The art mode button is an operation member to set the art mode, and when the art mode is set, a motion picture is captured (continuous shooting) while changing the photographing parameters following the still image photographing. The art mode may be set from a menu screen that is displayed on the display section 8 by the menu button without the need to provide the art mode button. Alternatively, it may also be possible to design so that the art mode is always set at the time of still image release without providing the art button.
The display section 8 has a display section, such as a liquid crystal panel and an organic EL panel, disposed on the backside etc. of the camera 10, like the display section 8 in the first embodiment, and produces a live view display before photographing, a normal playback display of a recorded image, a camera information display, etc. It is possible for a photographer to observe the live view display to determine a composition and timing. Further, the display section 8 in the present embodiment produces a playback display of an image captured in the art mode in which a motion picture is captured following the still image photographing.
The clock section 7 performs a timing operation and outputs information about the date and time of photographing. The information about the data and time of photographing is recorded at the same time when image data of a still image and motion picture is recorded in the recording section 6. When photographing in the art mode, the information is used to determine whether or not a predetermined period of time elapses after the photographing of a motion picture (continuous shooting) is started following the still image photographing.
The parameter setting section 3, the determining section 4, the recording section 6, and the flash ROM 12 are the same as the parameter setting section 3 etc. in the first embodiment, and therefore, their detailed description is omitted.
Next, the detailed configuration of the image processing section 5 will be described using
The temporary recording section 5a is a memory for temporarily recording image data from the imaging section 2 and has a memory 0 for temporarily recording image data of a still image and a memory 1 to a memory n for temporarily recording image data corresponding to each frame of a motion picture captured (continuous shooting) following the still image. The memory 0 for temporarily storing image data of a still image is connected to a still image compressing section 5b. The still image compressing section 5b performs compression of an image, such as JPEG
The memory 1 to the memory n for recording image data of each frame of a motion picture (continuous shooting) are connected to an order switching switch (SW) 5c. The order switching switch 5c switches the order of image data of each frame temporarily recorded in the memory 1 to the memory n. That is, after a motion picture is recorded, the last image data temporarily recorded in the memory n is read first, and then, image data is read in order of the memory n−1, the memory n−2, . . . , the memory 2, and the memory 1. Due to this, it is possible to play back an image as if time were reversed.
The order switching switch 5c is connected to an appropriate image selecting section 5d. The appropriate image selecting section 5d selects images of the image data temporarily recorded in the memory 1 to the memory n, which are suitable for a motion picture (continuous shooting), for example, images other than those in which the composition changes rapidly or blurred images. The appropriate image selecting section 5d is connected to an image duplicating section 5e. As will be described later, when photographing is completed, a motion picture captured for 0.5 seconds is expanded to two seconds and a playback display is produced. The image duplicating section 5e performs duplication while interpolating image data of each frame of the motion picture to expand the time of the image.
The image duplicating section 5e is connected to a motion picture compressing section 5g. The motion picture compressing section 5g performs motion picture compressing on the image data expanded to two seconds by the image duplicating section 5e with MPEG 4, motion JPEG etc. A connection data section 5f connects the image data of the motion picture generated by the motion picture compressing section 5g and the image data of the still image generated by the still image compressing section 5b and generates information about connection for continuously playing back both the image data. The information about connection of the connection data section 5f, the image data from the still image compressing section 5b, and the image data from the motion picture compressing section 5g are sent to the recording section 6 and recorded as one image file.
Next, photographing in the art mode by the camera 10 according to the present embodiment will be described using
An image 23b to an image 23d are images sequentially captured as a motion picture (continuous shooting) following the image 23a captured as a still image. The essential photographing is completed in the still image, and therefore, the photographing parameters determined by the photographer or determined automatically by the camera 10, the composition determined by the photographer, etc., may be invalidated and the image in the motion picture is more interesting because there is a motion therein. However, it is not easy to move the subject, and therefore, in the present embodiment, the photographing parameters are changed and thus the image is made dynamic.
In the examples shown in the image 23b to the image 23d, the effect of the change of photographing parameters is given, which is obtained by a close-up with an overexposure. As the operation to change the photographing parameters, it may also be possible to shift the focus position of the photographic lens. By shifting the focus position, such an effect that the face of a person becomes hazier can be obtained. In this case, this may be done along with the change in exposure correction value and zoom, or may be done alone. It may also be possible to blur the background by changing the aperture.
As described above, in the photographing in the art mode in the present embodiment, an image at a critical moment is captured in a still image and a motion picture is captured for a brief time after that. With the photographing in the art mode, it is possible for a user not having experienced photographing of a motion picture to easily enjoy dynamic motion picture photographing. With the photographing in the art mode, it becomes possible for a user to capture a varied picture without the need to manipulate the photographing parameters, such as the aperture.
Further, in the present embodiment, as will be described later, it is also possible to display how an image is formed on the display section 8 using image data acquired in the art mode after the photographing. If a user sees an image to which an effect by the art mode is given before photographing, it becomes difficult for the user to determine a composition and photo opportunity.
Next, a playback display produced by combining a plurality of images captured in the art mode (still image with a motion picture) described in
In an example shown in
At this time, it may also be possible to add an effect that seems to be liked by the person photographed instead of simply giving the transition effect at all times. For example, it may also be possible to add changes, for example, the exposure for a person A, the focus position for a person B, the zoom for a person C, and the aperture for a person D. It is of course possible to switch the transition effect to another by determining when the facial expression changes, or when the person blinks.
The image data used to produce a playback display as shown in
Next, a case is described using
That is, when photographing, a still image 26d is captured first and then, images 26c, 26b and 26a are captured as a motion picture in this order for 0.5 seconds. When the captured images are played back reversely, the motion picture is played back reversely for two seconds from the image 26a captured last to the images 26b and 26c in this order and finally, the image 26d captured as a still image is displayed as a critical piece.
Recently, an image sensor has been developed, which can capture images of hundreds of frames per second, and a playback in such a slow motion can be performed easily. Whether or not such a reverse playback is permitted can be determined by determining the change in facial expression of a subject and referring to the facial expression change DB 6a, and the reverse playback may be permitted only when the change in facial expression is not unnatural in the reverse playback. For example, in the case of a repetitive motion of, such as blinking, it may also be possible to arrange a motion picture in the reverse direction before a still image and a motion picture in the forward direction after the still image, and thus enabling a repetitive playback.
In the art mode in the present embodiment, after a still image is captured, a subject is also photographed in a motion picture (continuous shooting), and therefore, if the camera 10 is not directed toward the subject for a predetermined period of time after the still image photographing (0.5 seconds in the embodiment), an unsightly image may be obtained. Hence, in the present embodiment, the person 21 positioned in the center of the screen is extracted automatically and recorded with the face part of the person 21 being trimmed, in order not to apply a stress more than necessary to a photographer.
A trimming method will be described using
The image in
As described above, even if a photographer changes the compositions to the photographing regions 15a to 15c, the image data in the trimming regions 17b and 17c corresponding to the center part of the screen of the still image is recorded in the recording section 6. Even if a photographer feels easy after photographing a still image and unconsciously changes the compositions to the photographing regions 15b and 15c, the center part of the screen is trimmed and recorded. By trimming and using only the face part, it is possible to represent a motion picture for capturing the change in facial expression of the subject. That is, as shown in
Next, the operation of the camera 10 according to the present embodiment will be described using the flowchart of camera control shown in
When the mode is the photographing mode according to the result of the determination in step S201, an image is captured and displayed (S202). In this step, the image data acquired by the imaging section 2 is subjected to image processing in the image processing section 5 and a live view is displayed on the display section 8. It is possible for a photographer to determine a composition and photo opportunity while viewing the live view display.
After the image is captured and displayed, next, the screen features are determined (S203). In the screen feature determination, the face detecting section 2a detects whether or not the face part is included in the image and when included, detects the position and size thereof. When the face part is detected, the exposure control values, such as the aperture value and the shutter speed, are calculated so that the part achieves an appropriate exposure, and the automatic focus adjustment is made to focus the part.
After the screen feature determination is made, next, whether or not photographing is performed is determined (S204). Here, whether or not the release button of the operation section 9 is pressed and operated is determined. When photographing is to be performed according to the result of the determination, still image photographing is performed (S205). In this step, the image data output from the imaging section 2 is temporarily recorded in the memory 0 within the temporary recording section 5a of the image processing section 5 and at the same time, the image is compressed in the still image compressing section 5b.
After the still image photographing is performed, next, whether or not the mode is the art mode is determined (S206). The art mode is set by operating the art mode button of the operation section 9, and therefore, in this step, whether or not the art mode is set by operating the art mode button is determined. When the mode is not the art mode according to the result of the determination, the still image is recorded (S207). In this step, the image data of the still image acquired in step S205 and subjected to image compression processing is recorded in the recording section 6. After the still image is recorded, the process returns to step S201.
When the art mode is set according to the result of the determination in step S206, next, continuous photographing (motion picture) is started (S211). Here, continuous photographing (motion picture) to repeatedly acquire the image data from the imaging section 2 at a predetermined timing is started.
Subsequently, an exposure switch direction step determination is made (S212). When the art parameters are changed, to be described later, the photographing parameters are switched while performing continuous photographing, and at this time, it is necessary to change the aperture as a photographing parameter. In
After the exposure switch direction step determination is made, next, whether or not 0.5 seconds have elapsed is determined (S213). Here, the time from starting the continuous photographing in step S211 is measured by the clock section 11 and thus whether or not 0.5 seconds have elapsed is determined.
When 0.5 seconds have not elapsed according to the result of the determination in step S213, the art parameters are changed (S214). Here, the photographing parameters are changed while performing continuous photographing. In the present embodiment, a live view is displayed also during the continuous photographing, and therefore, it is possible to confirm the change in captured images accompanying the change of the photographing parameters. However, the change occurs in a brief time, and the change is too fast for the human eyes to notice, and therefore, 0.5 seconds are expanded to two seconds and the change is displayed like a slow motion. Details of the change of the art parameters will be described later using
When 0.5 seconds have elapsed according to the result of the determination in step S213, next, the continuous photographing is ended and a continuing display is provided (S215). Because 0.5 seconds have elapsed, the continuous photographing is ended, however, the time is expanded to two seconds in the change of the art parameters in step S214 and the (motion picture of) continuous shooting in the art mode is played back and displayed, and therefore, this playback display is continued.
Subsequently, whether or not there is a motion during the continuous shooting is determined (S216). Whether or not there is such a motion as described in
When there is a motion during the continuous shooting according to the result of the determination in step S216, the image is trimmed so that the part located in the center of the screen is centered (S217). Here, as described in
When the image is trimmed in step S217, or when there is not a motion during the continuous shooting according to the result of the determination in step S216, next, the images continuously shot are edited into a motion picture in the reverse direction while performing interpolation (S218). Here, as described using
Subsequently, the motion picture and the still image are recorded (S219). Here, data used to combine the motion picture (continuous shooting, images captured continuously) and the still image is attached by the connection data section 5f (refer to
After the motion picture and the still image are recorded, next, whether or not the time having elapsed from the previous recording is within a predetermined time is determined (S220). Here, whether or not the time from completing the previous continuous photographing (S215) to starting the next continuous photographing (S211) is within a predetermined time is determined. When the time is not within a predetermined time according to the result of the determination, the process returns to step S201.
When the time is within a predetermined time according to the result of the determination in step S220, next, connection with the previous motion picture is made (S221). As described using
It may also be possible to change the way of connection in accordance with the transition effect. For example, if a pure white overexposed screen is connected after a deep-black screen, it is not natural, and therefore, after the screen is transitioned to a deep-black one, it may also be possible to display a still image and then connect a motion picture in the forward direction toward overexposures rather than displaying the transition motion picture in the reverse direction from the overexposure side. After the connection with the previous motion picture is made in step S221, the process returns to step S201.
When the mode is not the photographing mode according to the result of the determination in step S201, next, whether or not the mode is the playback mode is determined (S231). The playback mode is set by pressing and operating the playback button of the operation section 9, and therefore, in this step, whether or not the playback mode is set is determined. When the mode is not the playback mode according to the result of the determination, the process returns to step S201.
When the mode is the playback mode according to the result of the determination in step S231, next, whether or not the mode is the art mode is determined (S232). In the recording section 6, not only the image data photographed by the normal still image photographing but also the image data photographed in the art mode is recorded. In this step, whether or not the currently selected image is captured in the art mode is determined based on the information attached to the image data.
When the mode is the art mode according to the result of the determination in step S232, the corresponding motion picture part is played back in a slow motion (S233). In the playback in the art mode, as described using
When the corresponding motion picture part is played back in a slow motion in step S233, or when the mode is not the art mode according to the result of the determination in step S232, next, a still image is played back (S234). In the case of a normal still image, the image selected in this step is enlarged to be played back and displayed on the display section 8. When the mode is the art mode, the auxiliary motion picture is played back in step S233 (corresponding to the images 26a to 26c in
As described above, in the example in the present embodiment, when photographing, immediately after the still image photographing, continuous shooting (motion picture photographing) is performed successively for five seconds as shown in
Next, the subroutine of changing art parameters in step S214 will be described using the flowchart shown in
Subsequently, a motion is detected and whether or not there is a motion is determined (S242, S243). Here, the motion determining section 4 determines whether or not there is camera shake or a change in composition, however, this is not limited and it may also be possible to make the determination based on a motion vector detected by the image processing section 5.
When there is a motion according to the result of the determination in step S243, next, whether or not a still image and an image are different is determined (S251). Here, the determination is made by comparing the still image captured by the still image photographing in step S205 with the image captured by the motion picture photographing in step S241. This is because if the difference between the still image and the motion picture is large, the continuity is impeded and it is not worth while viewing as a series of images. When the image is different from the still image according to the result of the determination in step S251, the process advances to step S215 and the continuous photographing is ended.
When there is not a motion according to the result of the determination in step S243, next, whether or not there is a room to change the aperture is determined (S244). Here, whether or not there is a room to open the aperture toward the fully open side from the currently set aperture value is determined.
When there is a room to change the aperture according to the result of the determination in step S244, next, the aperture is opened in such a way that the aperture is opened by one step for five frames (S245). The exposure is appropriate when a still image is captured, and however, when the aperture is opened gradually, the image gradually becomes an overexposed one. In the present embodiment, the aperture is opened by one step for five frames, however, this is an example, and the way may be changed appropriately. In the present embodiment, photographing of 30 frames per second is supposed, however, in the case of photographing of 60 frames per second, it may also be possible to open the aperture by one step for ten frames, and it may also be possible to appropriately set the number of frames for which the aperture is opened by one step in accordance with other conditions.
When there is not a room to change the aperture according to the result of the determination in step S244, whether or not there is a room to change the zoom is determined (S252). Here, the information about the current focal length of the zoom lens is acquired and then whether or not the focal length can be changed toward the longer focal length side is determined.
When there is a room to change the zoom according to the result of the determination in step S252, next, the focal length is sequentially increased by a factor of 1.1 for each frame (S253). The parameter control section 7 is instructed to increase the focal length by a factor of 1.1 each time it performs step S253. The factor of 1.1 is an example, and when, for example, high-speed continuous shooting is available, the focal length may be increased by a factor of 1.1 for two frames. The factor may be appropriately changed in consideration of the zooming speed, continuous shooting speed, etc.
When there is not a room to change the zoom according to the result of the determination in step S252, or when the still image and the image are not different according to the result of the determination in step S251, next, an exposure switch step is determined from the remaining time (S254). In this case, the exposure correction value is changed as a photographing parameter. A detailed flow of the exposure switch step determination will be described later using
Subsequently, the exposure is switched to another sequentially in accordance with the exposure correction value determined in step S254 (S255). Here, the aperture, the shutter speed, etc., are changed using an exposure correction value ΔE calculated in step S254.
When the aperture is opened sequentially in step S245, or when zooming in is performed sequentially in step S253, or when the exposure is switched to another sequentially in step S255, next, the images captured by photographing are displayed shifted from one another (S246). As described above, the continuous photographing (motion picture) is performed for 0.5 seconds, however, the time is too brief to recognize the transition effect by changing the photographing parameter, and therefore, a display is produced in a slow motion so that the display time is about two seconds. After the images captured by photographing are shifted, the process returns to the original flow.
Next, the subroutine of the exposure switch step determination in step S254 is described using the flowchart shown in
Subsequently, a difference E from the exposure of the still image is found (S262). Here, the difference E in the amount of exposure shifted from the amount of exposure at the time of the still image photographing in step S205 is calculated. Each time the subroutine of the exposure switch step determination is executed, the exposure is corrected and reduced by the amount of exposure E from that of the still image, and therefore, the reduced amount of exposure E is found here.
After the difference E from the exposure of the still image is found, next, an amount of change in exposure ΔE per unit time is found (S263). The amount of change in exposure ΔE is calculated by (three steps−E)/t. Here, the three steps is an aperture value, E is the difference in the amount of exposure found in step S262, and t is the remaining time of motion picture photographing found in step S261. In the present embodiment, the camera is designed so that when motion picture photographing is ended, the exposure correction by three steps is performed for the still image photographing and in step S263, the amount of change in exposure ΔE per unit time is found with which the exposure correction by three steps is performed when motion picture photographing is ended. Three steps is an example, and different values may be accepted in accordance with various conditions, such as the facial expression.
After the amount of change in exposure ΔE is found, next, an amount of exposure correction for each frame is found (S264). Here, it is found by dividing the amount of change in exposure ΔE for each frame by the number of frames in a unit time. After this calculation, the process returns to the original flow and the exposure is corrected in accordance with the amount of exposure correction ΔE (exposure correction value) for one frame.
Next, the subroutine of the exposure switch direction step determination in step S212 is described using the flowchart shown in
After the direction toward which there is a room to change the exposure is determined, next, whether there is a room to change exposure by three steps toward the overexposure side is determined (S272). Here, the determination is made based on the size of the room in step S271. When there is a room to change exposure by three steps toward the overexposure side according to the result of the determination, the exposure is transitioned toward the overexposure side (S273). On the other hand, when there is not a room to change exposure by three steps toward the overexposure side, next, the exposure is transitioned toward the underexposure side (S274).
When the transition toward the overexposure side is made in step S273 or when the transition toward the underexposure side is made in S274, the process returns to the original flow. When the transition toward the overexposure side is made, the amount of exposure correction is shifted toward the overexposure side and when the transition toward the underexposure side is made, the amount of exposure correction is shifted toward the underexposure side.
As described above, in the fourth embodiment of the present invention, when the art mode is set, image data is acquired as a still image and following this, while the photographing parameters are changed to ones different from those at the time of the photographing of the still image, image data is acquired by continuous shooting. Then, the image data of the still image and the continuous shooting acquired here is displayed in a representation different from the time flow with which images are captured by the continuous shooting at a timing in connection with the still image. Hence, it is possible to fuse the still image and the motion picture to capture and play back a varied image.
Further, in the fourth embodiment of the present invention, it is possible to capture an image to which the transition effect is attached, and therefore, it is possible to easily create a motion picture that combines a plurality of cuts that includes no rapid change in scene. Further, it is also possible to record a favorite image, which cannot be represented by a still image, as one image to which the effect of change is attached by changing the angle of view etc.
In the fourth embodiment of the present invention, as the photographing parameters, the aperture value, the focal length (zoom), and the exposure correction value are described, however, these are not limited, and for example, it may also be possible to sequentially switch the focus positions. When switching the focus positions at the time of still image photographing, for example, it may also be possible to gradually switch from the focus position where the main subject is in focus to the position where the background is in focus, or when there is another person other than the main subject, to the position where the other person is in focus. In particular, at the time of still image photographing, there is uncomfortable feeling if the subject is not in focus and it is also difficult to determine framing or a timing to photograph, and therefore, such specifications are reasonable that the photographing parameters are changed after photographing, as described in the fourth embodiment.
If the motion picture captured while changing the photographing parameters is played back in the reverse direction as shown in
In the fourth embodiment of the present invention, as the photographing parameters, the aperture value, the focal length (zoom), and the exposure correction value are described, however, it is acceptable if two of them or only one of them can be performed when all of the three cannot be performed. In the fourth embodiment of the present invention, the degree of blurring as a photographing parameter is changed by controlling the aperture value, however, it may also be possible to make use of image processing when blurring the background through image processing.
Further, in the fourth embodiment of the present invention, by simply changing the aperture and exposure correction value as the photographing parameters, an overexposure is achieved, however, this is not limited and it may also be possible to change the drive direction of the photographing parameters in accordance with a facial expression. For example, it may also be possible to underexpose when the facial expression becomes gloomy, or to increase the zooming speed when the facial expression becomes cheerful.
Furthermore, in the fourth embodiment of the present invention, continuous shooting (motion picture) is performed for 0.5 seconds and it is expanded to two seconds when played back and displayed, however, these values are examples, and other times may be accepted, of course. However, it is desirable for the time of continuous shooting (motion picture) to be such one which is not affected by any influence of camera shake. Other set values are examples, respectively, and values other than those set values may be accepted.
Still furthermore, in the fourth embodiment of the present invention, the image captured in the art mode is displayed on the display section 8 of the camera 10 immediately after the photographing or at the time of playback. The playback display may be produced, however, by a personal computer, for example, rather than the camera 10. In this case, the image data recorded in the recording section 6 may be played back and displayed as it is, and it may also be possible for the camera 10 not to perform image processing for reverse direction motion picture playback but to cause a personal computer to perform image processing for reversing the motion picture.
Next, a fifth embodiment of the present invention will be described using
Like the imaging section 2 in the first embodiment, the imaging section 2 includes a photographic lens (zoom lens) having a zoom function, an exposure control section, such as a shutter and aperture, an image sensor, a drive circuit and a read circuit of an image sensor, etc., and converts a subject image formed by the photographic lens into image data by the image sensor and outputs the data. Further, the photographic lens of the imaging section 2 includes a focus adjustment lens and its mechanism and in the present embodiment, it is also possible to photograph while shifting a focus by the focus control section 7b, to be described later, when photographing a transition motion picture. Furthermore, an aperture adjustment mechanism for blurring the background may be provided. The face detecting section 2a is the same as that in the first embodiment, and therefore, its detailed description is omitted.
The control section 1 controls the total sequence of the camera 10 in accordance with programs stored in the flash ROM 12. The control section 1 has the facial expression change extracting section 1a, the composition change determining section 1b, a contrast change extracting section 1d, a region dividing section 1e, and the image processing section 5. The facial expression change extracting section 1a is the same as that in the first embodiment, however, it may also be possible to cause the facial expression change extracting section 1a to predict the change in facial expression based on the speed of the change when the facial expression changes when determining the change in facial expression. Further, the composition change determining section 1b is the same as that in the first embodiment, and therefore, its detailed description is omitted.
The contrast change extracting section 1d extracts a high-frequency component (contrast) from image data output from the imaging section 2 and extracts the change in contrast. As will be described later, it is possible to obtain information about distance to subject by the contrast change extracting section 1d detecting a peak value of the change in contrast when scanning the photographic lens.
The region dividing section 1e divides the region of the screen based on the information about distance to subject obtained by the above-mentioned contrast change extracting section 1d. That is, the image data output from the imaging section 2 is divided into the form of a mesh and information about distance to subject is obtained at each point, and thus, it is possible to divide the screen region into each point having substantially the same distance to subject. The region division by the region dividing section 1e will be described later using
The image processing section 5 in the control section 1 has the motion picture section 5h and the still image section 5i, takes in image data output from the imaging section 2, performs various kinds of image processing, such as thinning processing, trimming processing, edge enhancement, color correction, and image compression, and performs image processing for displaying a live view of a motion picture, for recording a still image and a motion picture in the recording section 6, for playing back and displaying a still image and a motion picture, etc. For compression/decompression of an image etc., the image processing section 5 has circuits for a still image and a motion picture in accordance with each processing, such as a still image compressing/decompressing circuit and a motion picture compressing/decompressing circuit. Further, it is possible for the image processing section 5 to acquire color information and a motion vector based on image data.
The parameter control section 7 has the focus control section 7b. The focus control section 7b adjusts the focus position of the photographic lens of the imaging section 2 so that the contrast information detected by the contrast change extracting section 1d is indicative of the peak position. That is, a so-called automatic focus adjustment by contrast method is made. In the art motion picture mode, as will be described later, the focus adjustment of the photographic lens is made so that the focus moves from the secondary subject to the main subject.
The operation section 9 includes various operation members, such as the release button, the power switch, the playback mode setting button, and the menu button, and determines the operation state of these operation members and sends the determination result to the control section 1. The control section 1 described above controls photographing and playback in a predetermined sequence in accordance with the operation state of the operation member.
Like the display section 8 in the first embodiment, the display section 8 has a display section, such as a liquid crystal panel and an organic EL panel, disposed on the backside etc. of the camera 10 and produces a display of a live view before photographing, a display of normal playback of a recorded image, a display of camera information, etc. It is possible for a photographer to determine a composition and timing by observing the live view display. Further, the display section 8 in the present embodiment also plays back and displays a transition image obtained in the art motion picture mode, that is, by capturing a motion picture while moving the focus and finally photographing a still image.
The parameter setting section 3, the parameter control section 7, the motion determining section 4, the recording section 6, the facial expression change DB 6a, and the flash ROM 12 are the same as the parameter setting section etc. in the first embodiment, and therefore, their detailed description is omitted.
Next, the photographing in the art motion picture mode by the camera 10 according to the present embodiment is described using
As shown in
As described above, in the art motion picture mode in the present embodiment, image transition is performed, and in the transition from
Consequently, it is necessary to select a secondary subject and a main subject as a start point and an end point of the image transition. Either of the main subject and the secondary subject may be selected as the start point, however, in the present embodiment, the transition starts from the secondary subject and ends at the main subject because a completed still image is to be obtained finally. As a method of selecting a main subject and a secondary subject, there can be conceived of various methods, and in the present embodiment, a combination that exhibits the effect of focusing remarkably is preferred.
Next, the selection of the main subject and the secondary subject is described using
In the example shown in
As can be seen from
In the example shown in
When the screen is divided into each region, various pieces of information can be obtained for each region, such as whether a face is included, which region is wide, color components included in each region, and whether or not there is motion. By using these pieces of information, it is possible to determine which region is an important region, and therefore, to determine regions that include the main subject and the secondary subject described above. Further, when measuring the contrast while moving the focus position of the photographic lens after measuring the contrast between the focus positions Z1 to Z3 and dividing the screen into each region, if the contrast changes rapidly between the focus position Z3 and a focus position Z4, it is possible to modify the region and make use of the region in the determination as to whether the region is an important region by referring to an amount of change in contrast h and a width of change from the focus position Z2 where the face of the person 21a is focused at this time.
When there is a subject with high contrast (in the example in
Selection of a main subject and a secondary subject is determined based on weight values by weighting the divided regions, respectively, described in
For example, as to the first condition that “face with the upper half of the body”, whether or not the region 23a satisfies the condition is determined. In the example shown in
The third weighting condition that whether or not “there is a motion” in
The fifth and sixth weighting conditions relate to the size of the region area and are determined by the image processing section 5. When the area is large, it is considered that the region includes an important subject. The eighth weighting condition is that whether or not “screen center part” and this is determined by the image processing section 5. When a subject is in the screen center, this is done intentionally by a photographer in most cases and it can be considered that the subject is important.
As described above, the conditions are determined for each of the regions 23a to 23c and weight is assigned based on the determination result, the total of the points assigned is calculated, and then the region with the largest total value is determined to be the region Za in which the main subject is present. In the example shown in
When the region Za where the main subject is present is determined, next, the region a where the secondary subject is present is determined. When selecting the secondary subject, the values in either of the columns as shown in
As the conditions used in determining the region Zb where the secondary subject is present, the following conditions are added to the conditions for the main subject. That is, as the seventh weighting condition, whether or not “brighter than Za part” is determined. When the region a is brighter than the region Za of the main subject, this attracts attention of a viewer and the degree of importance can be thought high. Further, as the ninth weighting condition, whether or not “contrast is higher than face part” is determined. When the contrast is higher than the contrast of the face part in the image, there is a possibility that the degree of importance is high.
Furthermore, as the tenth weighting condition, whether or not “focus is distant from focus position” is determined. This is the case where the focus position is more than a predetermined distance apart from the main subject and in such a case, the effect of the focus shift turns out remarkably. This is important in selecting the secondary subject, and therefore, as high a point as seven is set in the present embodiment. In selecting the region Zb, the conditions shown in
Next, the operation of the camera 10 in the present embodiment is described using the flowchart of the art motion picture mode shown in
The flowchart shown in
When the release button is in the halfway-pressed state according to the result of the determination in step S301, the focus position is shifted (S302). Here, for example, the photographic lens is scanned between Z1 and Z3 in
Subsequently, an image is captured (S303). In step S302, the scanning of the photographic lens is started and while scanning, the contrast is acquired by the contrast change extracting section 1d using the image data from the imaging section 2 and information about distance to subject at each point is acquired.
Next, whether or not the measurement of a plurality of focuses is completed is determined (S304). Here, as described using
When the measurement of the plurality of focuses is completed according to the result of the determination in step S304, next, the screen is divided (S305). Here, as described in
When the region is divided, next, weighting of the region Za is performed (S306). Here, as described in
When the weighting of the region Za is performed, next, weighting of the region Zb is performed (S307). Here, as described in
When the weighting of the region a is performed, next, a difference in exposure dE between the Za region and the a region is determined (S308). Here, as to the Za region acquired in step S306 and the a region acquired in step S307, the luminance of the region is acquired using image data of each region and the difference in appropriate amount of exposure dE is calculated using the luminance.
When the difference in exposure dE is found, next, whether or not the release button is pressed fully is determined (S309). Here, whether or not the release button of the operation section 9 is in the fully-pressed state is determined. Let time elapse until the fully-pressed state is brought about and when the fully-pressed state is brought about, next, the focus and exposure are adjusted to the a region and photographing is performed (S311). The Zb region is the region of the secondary subject, and first, the focus and exposure are adjusted to this region. The state in this step S311 is the state of the image 25a illustrated in
Subsequently, a difference in focus dZ between the Za region and the Zb region is determined (S312). Here, the difference in focus dZ is found, which corresponds to a difference between information about distance to the main subject and information about distance to the secondary subject. The difference in focus dZ is used in step S316.
When the difference in focus dZ is determined, next, whether or not there is a motion in the a region is determined (S313). Here, whether there is something that moves in the region where the secondary subject is present is determined based on the motion vector of the image data.
When there is not a motion according to the result of the determination in step S313, let time elapse (S314). On the other hand, when there is a motion, let three seconds elapse. When there is a motion, a viewer desires to confirm what is the moving subject in many cases and hence, the waiting time is set long. One second and three seconds are examples, and the waiting time may be changed appropriately in consideration of the preferences of a photographer and a viewer.
After letting a predetermined time elapse in step S314 or S315, next, the focus shift is started toward the Za region and in this case, a speed in accordance with the difference in focus dZ is determined (S316). As described above, after the secondary subject is focused, the focus is moved toward the main subject and at this time, the moving speed is set to a value in accordance with the difference in focus dZ. Here, the focus is moved from the secondary subject to the main subject in a fixed time regardless of the difference in focus dZ. As a fixed time, for example, about three seconds are set, however, this may be changed appropriately in consideration of the preferences of a photographer and a viewer.
Subsequently, an image is captured and recorded while eliminating the difference in exposure dE (S317). In step S308, the difference in exposure dE is found and exposure is controlled so that an appropriate exposure is maintained in consideration of the difference in exposure dE during the focus shift. In order to control the exposure, the aperture value, the shutter speed, the ISO sensitivity, etc., are controlled. At this time, while the image data from the imaging section 2 is being subjected to image processing for a motion picture in the image processing section 5, the image data is recorded in the recording section 6 as a motion picture.
Next, whether or not the Za region is reached is determined (S318). The Za region is a region where the main subject is present and whether or not the focus position of the photographic lens reaches the region is determined. When the Za region is not reached according to the result of the determination, the process returns to step S317 and an image is captured and recorded while eliminating the difference in exposure dE.
On the other hand, when the Za region is reached according to the result of the determination in step S317, a still image is captured and recorded (S319). The Za region is the region of the main subject and when this region is reached, the image data from the imaging section 2 is subjected to image processing for a still image by the image processing section 5 and the image data of the still image is recorded. The state in step S319 corresponds to the image 25b illustrated in
In the flowchart shown in
As described above, in the fifth embodiment of the present invention, the screen is divided into a plurality of regions in accordance with the subject and a region from which the focus is moved is determined from the plurality of regions and the focus is moved by the focus control section. Hence, it is possible to capture a varied image without the need of a photographer to perform a manual operation intentionally. Further, the focus control section is configured to record image data of a motion picture when moving the focus, and therefore, it is possible to easily perform motion picture photographing.
Further, in the fifth embodiment of the present invention, the focus control section performs motion picture photographing while moving the focus and performs still image photographing when the target focus position is reached. Hence, it is possible to easily capture a varied image that fuses a motion picture and a still image together. The motion picture obtained by such photographing exhibits a very significant effect without the presence of the still image and will serve as an effective raw material when only the motion picture parts are connected and edited.
Furthermore, in the fifth embodiment of the present invention, when the subject image includes the main subject and first and second image parts other than the main subject, the direction of the movement of the focus is determined using the difference in focus between the main subject and the first image part and the difference in focus between the main subject and the second image part. Specifically, when determining the secondary subject, the fact that the focus is apart from the position of focus adjustment (10) in
In the fifth embodiment of the present invention, when information about distance to subject is acquired, the contrast method is used, in which the high-frequency component of image data is extracted, however, this is not limited and other ranging methods may be used. For example, a difference in phase AF may be employed and in this case, it is no longer necessary to scan the photographic lens.
Further, in the fifth embodiment of the present invention, when the screen is divided, information about distance to subject is used, however, this is not limited and other methods may be used. For example, it may also be possible to analyze the image and extract the contour, then, divide the screen based on the contour.
Furthermore, in each of the embodiments of the present invention described above, explanation is given using a digital camera as a device for photographing, however, the camera may be a digital single-lens reflex camera or a compact digital camera, a camera for motion picture photographing, such as a video camera and a movie camera, or a camera incorporated in a mobile telephone, personal digital assist (PDA) terminal, gaming machine, etc.
The present invention is not limited to the above-mentioned embodiments as they are but can be embodied by modifying components without deviating from the scope of the gist in the embodiment stage. Further, it is also possible to form various inventions by appropriately combining the plurality of components disclosed in the above-mentioned embodiments. For example, it may also be possible to delete some components of the entire components shown in the embodiments. Furthermore, it may also be possible to appropriately combine the components across the different embodiments.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2009-141902 | Jun 2009 | JP | national |
| 2009-141903 | Jun 2009 | JP | national |
| 2009-141904 | Jun 2009 | JP | national |
This application is a divisional application of U.S. patent application Ser. No. 12/815,082 (referred to as “the '082 application” and incorporated herein by reference), filed on Jun. 14, 2010, titled “PHOTOGRAPHING DEVICE, PHOTOGRAPHING METHOD, AND PLAYBACK METHOD,” and listing Keiji KUNISHIGE, Satoshi MIYAZAKI, Masaomi TOMIZAWA, and Osamu NONAKA as inventors, the '082 application claiming benefit, under 35 U.S.C. §119, to the filing date of prior Japanese Patent Applications Nos. 2009-141902, 2009-141903, 2009-141904 filed on Jun. 15, 2009. These applications are expressly incorporated herein by reference. The scope of the present invention is not limited to any requirements of the specific embodiments described in the applications.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7319480 | Akiyama | Jan 2008 | B2 |
| 7626613 | Kawamura | Dec 2009 | B2 |
| 20060238620 | Asada | Oct 2006 | A1 |
| 20080024620 | Yoshida | Jan 2008 | A1 |
| Number | Date | Country |
|---|---|---|
| 2009-060471 | Mar 2009 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20140240530 A1 | Aug 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12815082 | Jun 2010 | US |
| Child | 14192203 | US |
