Patent Application
20250240399
This application claims the benefit of Japanese Patent Application No. 2024-007917, filed on Jan. 23, 2024, and Japanese Patent Application No. 2024-092912, filed on Jun. 7, 2024, each of which is hereby incorporated by reference herein in its entirety.
The present invention relates to an electronic device and a control method of the electronic device.
In general, a VR image (VR video) includes two images (a left-eye image and a right-eye image) that can be viewed stereoscopically by a user by using a parallax when the VR image (VR video) is reproduced on a head-mounted display (HMD). For example, the VR image includes two equirectangular projection images resulting from equirectangular conversion performed on circular fisheye images captured through two circular fisheye lenses.
When mounting of the two circular fisheye lenses is not appropriate, a parallax between the two circular fisheye images may not be appropriate for stereoscopy. In this case, by adjusting parameters to be used for the equirectangular conversion by which the circular fisheye images are converted to the equirectangular projection images, it is possible to achieve conversion to the equirectangular projection images such that the parallax is appropriate for stereoscopy.
However, there may be a case where, as a result of the parameter adjustment, each of the circular fisheye images has no valid region corresponding to a local region of the equirectangular projection image, and the local region of the equirectangular projection image is set to an invalid region having invalid pixel values.
In Japanese Patent Application Publication No. 2022-184139 discloses a technology which provides, during equirectangular conversion of a VR image, an invalid region of each of left and right images with the same shape.
There is a method in which, in a case of generating a video file of VR images, parameters to be used for equirectangular conversion are adjusted so as to correct a motion blur between a frame of concern and a frame previous or subsequent thereto and thereby carry out camera shake correction. As described above, depending on values of the parameters to be used for the equirectangular conversion, a region of the equirectangular projection image resulting from the conversion may locally have an invalid region. When the parameters to be used for the equirectangular conversion differ from one frame to another, it may be possible that, by performing the camera shake correction, the invalid region of the equirectangular projection image moves during video reproduction.
As a result, when the camera shake correction is performed, flickering is observed in the equirectangular projection image. However, the technology described in Japanese Patent Application Publication No. 2022-184139 cannot suppress the flickering.
It is, therefore, an object of the present invention to be able to generate a VR video with reduced flickering during reproduction.
An aspect of the disclosure is an electronic device that converts a first video in a first display mode including a shooting region representing a result of shooting by an image capturing apparatus to a second video, which is a video in a second display mode including a plurality of frames, the electronic device including one or more processors and/or circuitry configured to: perform acquisition processing of acquiring the first video; perform setting processing of setting invalid regions of the same shape to the same positions in the plurality of frames of the second video; and perform generation processing of converting the first video so as to set a pixel value representing invalidity to each of the invalid regions in the plurality of individual frames of the second video and thereby generating the second video.
An aspect of the disclosure is a control method of an electronic device that converts a first video in a first display mode including a shooting region representing a result of shooting by an image capturing apparatus to a second video, which is a video in a second display mode including a plurality of frames, the control method including: an acquisition step of acquiring the first video and meta data related to a shooting state of the first video; an information generation step of generating, on a basis of the meta data, correction information related to geometrical correction to be performed during generation of the second video; a mask generation step of generating a mask image covering a local region of the second video, the mask image being applied to each of the frames; and a generation step of generating, on the basis of the correction information, the second video through conversion of a first video.
Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Referring to the accompanying drawings, the following will describe embodiments of the present invention in detail.
The editing apparatus 100 includes a control unit 101, a ROM 102, a RAM 103, an external storage apparatus 104, an operation unit 105, a display unit 106, a communication unit 107, and a system bus 108.
The control unit 101 controls the entire editing apparatus 100. For example, the control unit 101 is a Central Processing Unit (CPU). The control unit 101 can also perform image generation, setting processing, and calculation processing. Accordingly, the control unit 101 can also operate hereinbelow as a “generation unit”, a “setting unit”, and a “coordinate calculation unit”.
The ROM 102 is a Read Only Memory (ROM) which stores information (programs and parameters) that need not be changed.
The RAM 103 is a Random Access Memory (RAM) which temporarily stores information (programs and data) supplied from an external apparatus or the like.
The external storage apparatus 104 is disposed by being fixed to the editing apparatus 100. The external storage apparatus 104 is a hard disk or a flash memory. Alternatively, the external storage apparatus 104 includes an optical disk detachable from the editing apparatus 100, such as a floppy disk (FD) or a Compact Disk (CD). The external storage apparatus 104 may also have a magnetic card, an optical card, an IC card, a memory card, or the like.
The operation unit 105 receives an instruction according to an operation of a user. The operation unit 105 includes an operation member (a button or a touch panel) operable by the user.
The display unit 106 displays data held by the editing apparatus 100 and data supplied to the editing apparatus 100.
The communication unit 107 communicates with an external apparatus such as a camera (image capturing apparatus such as a digital camera).
The system bus 108 provides communicable connection between the individual components (units).
A video file including image data, meta data, and the like is stored (recorded) in the external storage apparatus 104. The video file can also be read from the external storage apparatus 104. The communication unit 107 can also receive the video file from an external apparatus such as a camera. Therefore, in the following, the “input unit” or the “acquisition unit” for data is a general name of the external storage apparatus 104 and the communication unit 107.
Referring to the flow chart of
In Step S201, the camera determines whether or not sensor data during the shooting can be stored (recorded) in the video file. The sensor data mentioned herein is inertia data representing, e.g., an acceleration measured by a gyro sensor or the like. The camera periodically stores the inertial data from the gyro sensor in the video file to be able to record movement of the camera during the shooting. When it is determined that the sensor data can be stored, a flow advances to Step S202. When it is determined that the sensor data cannot be stored, the flow advances to Step S203.
In Step S202, the camera stores the sensor data as timed meta data (meta data having information in time series) in the video file.
In Step S203, the camera determines whether or not audio (sound and voice) is to be stored in the video file. Depending on a shooting mode, the control unit 101 may not store the audio in the video file. When it is determined that the audio is to be stored, the flow advances to Step S204. When it is determined that the audio is not to be stored, the flow advances to Step S205.
In Step S204, the camera stores the audio data (sound/voice data) as audio sampling data in the video file.
In Step S205, the camera stores an image in one frame as video sampling data in the video file.
In Step S206, the camera determines whether or not an instruction to end the shooting has been given. When it is determined that the instruction to end the shooting has been given, processing in the present flow chart is ended. When it is determined that the instruction to end the shooting has not been given, the processing in the present flow chart returns to Step S201.
The video header region 301 holds information related to the entire video (hereafter referred to as “video-related information”). For example, the video-related information includes at least any of information items such as an image size (a width and a height) of a moving image, a frame rate, an audio sample frequency, a sample size, and the number of channels. The video-related information may also include information items such as a position (a reproduction position or a reproduction time) of a sample (which is data in one frame in a case of the video) of each medium (such as the video, the audio, or the timed meta data) and a size thereof.
The file meta data region 302 holds meta data related to a file and the entire video. For example, in the file meta data region 302, information related to a lens during the shooting, a design value (such as a center portion or radius of an image circle), a measurement value resulting from measurement based on calibration, and the like are stored as the meta data.
The sampling data region 303 stores sampling data of each medium. The sampling data region 303 includes a video sampling region 3031, an audio sampling region 3032, and a timed meta data sampling region 3033.
The video sampling region 3031 stores the sampling data of the video medium. The sampling data of the video medium is image data stored in Step S205 in
The audio sampling region 3032 stores the sampling data of the audio medium. The sampling data of the audio medium is audio data stored in S204 in
The timed meta data sampling region 3033 stores the sampling data of the timed meta data medium. For example, the sampling data of the timed meta data medium is the sensor data stored in Step S202 in
A circular fisheye image 401 is an example of one circular fisheye image recorded by the camera with the fisheye lens mounted therein.
An image circle 4011 is a region (shooting region) where an object to be imaged is captured by the camera via the fisheye lens. Inside of the image circle 4011, the imaged object is recorded. The outside of the image circle 4011 is an invalid region where no image is observed, and black or a color close to black is recorded. The invalid region is a region irrelevant to a result of the shooting. Positional information (information such as a center position and a radius) of the image circle 4011 in the circular fisheye image is stored as file meta data in the file meta data region 302. Normally, the position of the image circle 4011 is constant since a position of the lens does not vary during the shooting.
A region 4012 is a region to be subjected to equirectangular conversion. The inside of the region 4012 is developed in the equirectangular image. Information representing a size of the region 4012 is also stored as the file meta data in the file meta data region 302. The position of the region 4012 has the same center as the center of the image circle 4011 in default, but can also be changed.
An equirectangular image 402 is an equirectangular image resulting from the equirectangular conversion of the region 4012 included in the image circle 4011.
A circular fisheye image 403 is an example of the circular fisheye image recorded by the camera, similarly to the circular fisheye image 401.
An image circle 4031 is an image circle, similarly to the image circle 4011. A region 4032 is a region to be subjected to the equirectangular conversion, similarly to the region 4012.
In the circular fisheye image 403, the camera is oriented slightly more leftward than in the circular fisheye image 401, and accordingly the imaged object is observed at a position slightly shifted to a right side from a center inside of the image circle 4031.
The equirectangular image 404 is obtained as a result of performing the equirectangular conversion on the region 4032 included in the circular fisheye image 403. In the circular fisheye image 403, a position of the region 4032 with respect to the image circle 4031 is the same as a position of the region 4012 with respect to the image circle 4011 in the circular fisheye image 401. Accordingly, in the equirectangular image 404, the imaged object is placed at a position slightly shifted to the right side from the center of the image.
A circular fisheye image 405 is an example of the circular fisheye image recorded by the camera, similarly to each of the circular fisheye image 401 and the circular fisheye image 403.
An image circle 4051 is an image circle, similarly to each of the image circle 4011 and the image circle 4031.
A region 4052 is a region to be subjected to the equirectangular conversion, similarly to each of the region 4012 and the region 4032. A position of the region 4052 is slightly shifted to the right side compared to the positions of the region 4012 and the region 4032.
An equirectangular image 406 is an equirectangular image obtained as a result of the equirectangular conversion of the region 4052 included in the circular fisheye image 405. A position of the region 4052 is located closer to the right side of the image circle 4051, and therefore the imaged object is located at the center in the equirectangular image. Thus, in each of the following embodiments, when the equirectangular image is to be generated from the circular fisheye image, a region of the circular fisheye image to be subjected to the equirectangular conversion is changed. As a result, it is possible to obtain an effect of reducing an effect of camera shake in the equirectangular image.
Meanwhile, the right side of the region 4052 locally spreads out from the outside (which equals to the invalid region) of the image circle 4051, and accordingly a local region of the right side of the equirectangular image 406 obtained as a result of the equirectangular conversion refers to the invalid region of the circular fisheye image 405. In the invalid region, there is no valid pixel value of the circular fisheye image 405, and therefore a pixel (e.g., black-color pixel) having a pixel value representing the invalid region is present in the equirectangular image 406.
Referring to a flow chart of
In Step S501, the control unit 101 reads, through the input unit, common meta data (meta data common to all the frames) stored in the file meta data region 302 of the video file therefrom. For example, the common meta data includes information items such as the center position of the image circle, the radius of the image circle, and a radius of a region to be subjected to the equirectangular conversion. The control unit 101 also reads, as the meta data, parameter values (manual correction parameter values) set by the user by using the operation unit 105. These meta data items are used when the equirectangular image is to be generated from the circular fisheye image in the subsequent step.
In Step S502, the control unit 101 calculates the shake correction value for each of all the frames. Processing in Step S502 will be described in detail with reference to the flow chart of
In Step S503, the control unit 101 calculates the invalid region of the equirectangular image in each of all the frames. Processing in Step S503 will be described in detail with reference to the flow chart of
In Step S504, the control unit 101 generates the equirectangular image in each of all the frames, and outputs the generated equirectangular image. Processing in Step S504 will be described in detail with reference to the flow chart of
Referring to the flow chart of
In Step S601, the control unit 101 sets, to 0, each of a variable i representing a frame number of the read image data and a variable n representing a frame number of the image data for which the shake correction value is to be calculated.
In Step S602, the control unit 101 determines whether or not the image data (image data of the circular fisheye image) in each of all the frames stored in the video file has been read. When it is determined that the image data in each of all the frames has been read, a flow advances to Step S612. When it is determined that the image data in at least one of the frames has not been read, the flow advances to Step S603.
In Step S603, the control unit 101 reads the image data of the circular fisheye image in the frame having a frame number i (which equals to the frame i) from the video sampling region 3031 of the video file. When the image data stored in the video sampling region 3031 is compressed herein, the control unit 101 decodes the image data to acquire the image data in a RGB format.
In Step S604, the control unit 101 analyzes the read image data (performs image processing on the image data) to extract feature points in the circular fisheye image.
In Step S605, the control unit 101 determines whether or not the sensor data in the frame i is stored in the timed meta data sampling region 3033 of the video file. When it is determined that the sensor data in the frame i is stored, the flow advances to Step S606. When it is determined that the sensor data in the frame i is not stored, the flow advances to Step S607.
In Step S606, the control unit 101 reads, from the timed meta data sampling region 3033, the sensor data in the frame i (sensor data related to motion of the camera during shooting of the circular fisheye image in the frame i).
In Step S607, the control unit 101 adds 1 to the variable i representing the read frame number.
In Step S608, the control unit 101 determines whether or not the image data in the number of frames required to calculate a shake amount in the frame of the frame number n (which equals to the frame n) has been read. For example, the shake amount is an “amount of camera movement or amount of movement of the object to be imaged in the image data” due to cameral shake during a specific period around a time of one frame. Consequently, to calculate, e.g., the shake amount in one frame, the image data in each of 1-second frames previous and subsequent to the one frame is required. When it is determined that the image data in the required number of frames has been read, the flow advances to Step S609. When it is determined that the image data in the required number of frames has not been read, the flow returns to Step S602.
In Step S609, the control unit 101 calculates the shake amount in the frame n. For example, the control unit 101 calculates the shake amount in the frame n on the basis of “information on the feature points and the sensor data” in each of the frame n and the frames previous and subsequent thereto. At this time, the control unit 101 may also calculate the shake amount on the basis of only the information on the feature points or on the basis of only the sensor data.
The motion of the camera due to camera shake unintended by the user is different from movement of the camera as camerawork. For example, in the event of camera shake, the camera performs lateral or vertical reciprocal movement at short periods. Accordingly, the control unit 101 can calculate the shake amount on the basis of a movement pattern of the feature points between the plurality of frames or a value of a gyro sensor mounted in the camera.
In Step S610, the control unit 101 calculates a shake correction value for correcting shake in the shake amount on the basis of the shake amount in the frame n calculated in Step S609.
In Step S611, the control unit 101 adds 1 to the variable n.
As described above, to calculate the shake correction amount in one frame, information on the frames previous and subsequent thereto is also required. Accordingly, at a time when all the frames are read, the shake correction value for each of all the frames has not been calculated yet, and the plurality of frames for which the shake correction values are uncalculated remain. Consequently, in Step S612 and thereafter, the shake correction values for these remaining fames are calculated.
In Step S612, the control unit 101 determines whether or not the shake correction value for each of all the frames has been calculated. When it is determined that the shake correction value for each of all the frames has been calculated, the processing in the present flow chart is ended. When it is determined that the shake correction value for each of all the frames has not been calculated yet, the flow advances to Step S613.
In Step S613, the control unit 101 calculates the shake amount in the frame n. Processing in Step S613 is the same processing as the processing in Step S609.
In Step S614, the control unit 101 calculates, on the basis of the shake amount in the frame n calculated in Step S613, the shake correction value for correcting the shake in the shake amount. Processing in Step S614 is the same processing as the processing in Step S610.
In Step S615, the control unit 101 adds 1 to the variable n.
Referring to a flow chart of
In Step S701, the control unit 101 sets the variable i representing the frame number to 0.
In Step S702, the control unit 101 determines whether or not calculation of the invalid pixels in the equirectangular image in each of all the frames has been ended. The invalid pixels mentioned herein are pixels to which invalid pixel values (specific pixel values) are to be set in the subsequent step. When it is determined that the calculation of the invalid pixels in the equirectangular image in each of all the frames has been ended, the processing in the present flow chart is ended. When it is determined that the calculation of at least any of the invalid pixels in the equirectangular image in each of all the frames has not been ended, a flow advances to Step S703.
In Step S703, the control unit 101 acquires the shake correction value for the frame i. The shake correction value is used to correct a position of a region to be subjected to the equirectangular conversion in the circular fisheye image 405 illustrated in
In Step S704, the control unit 101 sets coordinates (x, y) of a pixel to be subjected to processing in the equirectangular image in the frame i to (0, 0). The control unit 101 processes both one-eye images (both of a left-eye image and a right-eye image) in the equirectangular image in one loop (Steps S705 to S710). In each of the one-eye images in a VR image, a width and a height are equal to each other. Accordingly, when it is assumed that a width of each of the one-eye images is j pixels and a height of the one-eye image is j rows, coordinates of the one-eye image are included in a range of a top left (0, 0) to a right bottom (j-1, j-1) of the one-eye image.
In Step S705, the control unit 101 determines whether or not processing in Steps S706 to S710 has been performed on each of all the pixels in the equirectangular image in one frame. When it is determined that the processing in Steps S706 to S710 has been performed on each of all the pixels, the flow advances to Step S711. When it is determined that the processing in Steps S706 to S710 has not been performed on at least any of all the pixels, the flow advances to Step S706.
In Step S706, the control unit 101 obtains coordinates (xL, yL) of the circular fisheye image corresponding to coordinates (x, y) of a pixel to be processed in the left-eye image. The coordinates in the circular fisheye image corresponding to the coordinates in the equirectangular image can be calculated by performing conversion inverse to the equirectangular conversion. At this time, the control unit 101 determines the coordinates (xL, yL) in the circular fisheye image on the basis of the common meta data read in Step S501 and manual correction parameter values set by the user. The control unit 101 also determines the coordinates (xL, yL) in the circular fisheye image on the basis of the shake correction value acquired in Step S703 in consideration of shake correction in the equirectangular image.
In Step S707, the control unit 101 determines coordinates (xR, yR) in the circular fisheye image corresponding to the coordinates (x, y) of a pixel to be processed in the right-eye image. This is the same processing as that in Step S706.
In Step S708, the control unit 101 determines whether or not both of the two coordinates (xL, yL) and (xR, yR) belong to the inside of a valid region. In the first embodiment, the valid region is the image circle (shooting region). When it is determined that the both coordinates are coordinates in the image circle, the flow advances to Step S710. When it is determined that at least either one of the both coordinates is not in the image circle, the flow advance to Step S709.
In Step S709, the control unit 101 determines that a pixel in the equirectangular image to be processed is the invalid pixel, and stores the coordinates (x, y) of the pixel in an invalid coordinate table. The invalid coordinate table is used commonly to all the frames of the VR video. Accordingly, in the invalid coordinate table, coordinates of pixels determined to be the invalid pixels in at least any of the frames are registered. Therefore, by thus controlling the invalid coordinate table, the control unit 101 can set, as the invalid region, a set of the coordinates (invalid coordinates) of the pixels determined to be the invalid pixels in at least any of the frames.
In Step S710, the control unit 101 updates the coordinates (x, y) to coordinates of a next pixel. The coordinates of the next pixel are coordinates obtained by adding 1 to x, which is an X-coordinate. However, when the X-coordinate obtained by adding 1 to x reaches the width j, the coordinates of the next pixel are coordinates obtained by adding 1 to y, which is a Y-coordinate, and returning x, which is the X-coordinate, to 0.
In Step S711, the control unit 101 adds 1 to the variable i representing the frame number.
Referring to a flow chart of
In Step S801, the control unit 101 sets the variable i representing the frame number to 0.
In Step S802, the control unit 101 determines whether or not the equirectangular image in each of all the frames included in the VR video has been output. When it is determined that the equirectangular image in each of all the frames has been output, the processing in the present flow chart is ended. When it is determined that the equirectangular image in at least any of all the frames has not been output, the flow advances to Step S803.
In Step S803, the control unit 101 reads the image data of the circular fisheye image in the frame i from the video sampling region 3031 through the input unit. When the image data stored in the video sampling region 3031 is compressed, the control unit 101 decodes the image data to convert the image data to an RGB format.
In Step S804, the control unit 101 acquires the shake correction value for the frame i. Processing in Step S804 is the same processing as the processing in Step S703.
In Step S805, the control unit 101 sets the coordinates (x, y) of the pixel to be processed in the equirectangular image to (0, 0). Similarly to the processing in the flow chart of
In Step S806, the control unit 101 determines whether or not the pixel values of all the pixels in the equirectangular image in the frame i have been set. When it is determined that the pixel values of all the pixels in the equirectangular image in the frame i have been set, the flow advances to Step S813. When it is determined that the pixel value of at least any of all the pixels in the equirectangular image in the frame i has not been set, the flow advances to Step S807.
In Step S807, the control unit 101 determines whether or not the coordinates (x, y) of the pixel to be processed have been registered in the invalid coordinate table (i.e., the pixel at the coordinates (x, y) is included in the invalid region). When it is determined that the coordinates (x, y) have been registered in the invalid coordinate table, the flow advances to Step S808. When it is determined that the coordinates (x, y) have not been registered in the invalid coordinate table, the flow advances to Step S810.
In Step S808, the control unit 101 sets a pixel value indicating invalidity (invalid pixel value) to the pixel value at the coordinates (x, y) in the left-eye image in the equirectangular image. For example, the pixel value indicating invalidity is a specific pixel value such as a pixel value representing black, a pixel value representing white, or a pixel value representing blue.
In Step S809, the control unit 101 sets the pixel value indicating invalidity to the pixel value at the coordinates (x, y) in the right-eye image in the equirectangular image. According to Steps S808 and S809, in the left-eye image and the right-eye image in the equirectangular image, the pixel value indicating invalidity is set to each of the invalid regions at the same position and having the same shape.
In Step S810, the control unit 101 sets, on the basis of the shake correction value, the pixel value at the coordinates (x, y) in the left-eye image in the equirectangular image. In Step S811, the control unit 101 sets, on the basis of the shake correction value, the pixel value at the coordinates (x, y) in the right-eye image in the equirectangular image. Thus, the pixel value of the pixel in the equirectangular image is set on the basis of the shake correction value to allow the equirectangular image with a reduced effect of the camera shake to be generated. Accordingly, in a case where the object to be imaged is not moving and camera shake is observed, in a plurality of consecutive frames, positions in the equirectangular image at which the imaged object is seen can be maintained at a given position. Details of the processing in Steps S810 and S811 will be described later with reference to the flow chart of
In Step S812, the control unit 101 updates the coordinates (x, y) to the coordinates of the next pixel. The coordinates of the next pixel are coordinates obtained by adding 1 to x, which is the X-coordinate. However, when the X-coordinate obtained by adding 1 to x reaches the width j, the coordinates of the next pixel are coordinates obtained by adding 1 to y, which is the Y-coordinate, and returning x, which is the X-coordinate, to 0. Note that, by repeating the processing in Steps S807 to S812 for each of the pixels in the equirectangular image in one frame, the pixel value of each of the pixels in the equirectangular image in the frame is set and, consequently, the equirectangular image in the frame is generated.
In Step S813, the control unit 101 outputs the equirectangular image in the frame number i, which is generated in Step S803 to Step S812. In a case where the equirectangular image is stored in a file, the equirectangular image is output to the external storage apparatus 104. In a case where the equirectangular image is displayed, the equirectangular image is output to the display unit 106.
In Step S814, the control unit 101 adds 1 to the variable i representing the frame number.
Referring to the flow chart of
In Step S901, the control unit 101 calculates, on the basis of the shake correction value or the like, coordinates (X, Y) of the circular fisheye image corresponding to the coordinates (x, y) in the equirectangular image. The processing in Step S901 is the same processing as the processing in Steps S706 and S707 in
In Step S902, the control unit 101 acquires the pixel value at the coordinates (X, Y) of the circular fisheye image. Alternatively, the control unit 101 acquires, as the pixel value at the coordinates (X, Y), a pixel value obtained by performing interpolation using a bicubic method or the like on the basis of the number of the pixels around the coordinates (X, Y) calculated in Step S901.
In Step S903, the control unit 101 sets the pixel value acquired in Step S902 as a pixel value at the coordinates (x, y) in the equirectangular image.
According to the first embodiment, in a case where shake correction differing from one frame to another is performed, even when the equirectangular image is generated from the circular fisheye image, it is possible to provide each of the invalid regions formed in the equirectangular image with the same position and the same shape throughout all the frames. This can suppress flickering caused by changes in the shape and position of the invalid region during reproduction of the VR video (equirectangular image).
In
The glass portion 1011 is a glass portion of a left-eye fisheye lens. The support 1012 is a support for the glass portion 1011 of the left-eye fisheye lens.
The glass portion 1013 is a glass portion for a right-eye fisheye lens. The support 1014 is a support for the glass portion 1013 of the right-eye fisheye lens. The support 1015 is a support for the left and right two fisheye lenses.
An angle of view 1016 connecting a line segment AB and a line segment BC is an angle of view of a shooting range of the left-eye fisheye lens. In the left-eye fisheye lens, the angle of view 1016 is a 180-degree angle of view. An angle of view 1017 connecting line segments DE and EF is an angle of view of a shooting range of the right-eye fisheye lens. Similarly to the angle of view 1016, the angle of view 1017 is a 180-degree angle of view.
A circular fisheye image 1020 is an example of a circular fisheye image acquired by the camera with the double fisheye lens 1010 mounted therein.
An image circle 1021 is an image circle corresponding to the glass portion 1011. An image circle 1022 is an image circle corresponding to the glass portion 1013.
When the positions of the two lenses are close to each other and the angle of view of the shooting range of each of the lenses is close to 180 degrees as in the double fisheye lens 1010, into a shooting range using one of the lenses, another of the lenses goes.
An image 1023 is an image of the glass portion 1013 reflected on the glass portion 1011. Likewise, an image 1024 is an image of the glass portion 1011 reflected on the glass portion 1013. These images are in the image circles, and can therefore be subjected to equirectangular conversion. In addition, due to parameter values for camera shake correction (parameter values for the equirectangular conversion), positions in an equirectangular image where these images are reflected are not constant. Meanwhile, when the two fisheye lenses have a fixed positional relationship therebetween as in the double fisheye lens 1010, the positions in the circular fisheye image where these images are reflected are constant.
Referring to
In Step S1101, the control unit 101 acquires information related to an excluded region specified in advance in the image circle of the circular fisheye image. The information related to the excluded region is, e.g., information related to coordinates in images (the image 1023 and the image 1024) which are not needed as the VR images in the image circle of the circular fisheye image 1120, as illustrated in
In Step S1110, the control unit 101 determines whether or not each of the two coordinates (xL, yL) and (xR, yR) of the circular fisheye image, which are obtained in Step S706 and S707, is a coordinate outside of the excluded region. When it is determined that both coordinate pairs are coordinates outside of the excluded region, the flow advances to Step S710. When it is determined that at least one of the coordinates is a coordinate inside of the excluded region, the flow advances to Step S709.
According to the second embodiment, coordinates in the excluded region inside of the image circle of the circular fisheye image are also regarded as a region other than the valid region, similarly to the region outside of the image circle. In other words, the region in the equirectangular region corresponding to the excluded region is also regarded as the invalid region. Accordingly, an unneeded image corresponding to the excluded region is not displayed in the equirectangular image and the shape of the invalid region does not vary from one frame to another, and therefore it is possible to suppress flickering of the VR video and improve a sense of immersion of the user in the VR video.
In each of the first and second embodiments, to calculate the invalid region, calculation in a direction inverse to that of the equirectangular conversion is performed. It can be considered that such processing involves a large amount of calculation, and requires a long processing time. However, depending on cases, it is desired to calculate the invalid region of the equirectangular image within a short processing time, though accuracy of the invalid region may be low. Therefore, in a third embodiment, a description will be given of the editing apparatus 100 capable of calculating the invalid region of the equirectangular image within a short processing time by calculating the invalid region depending on shaking of the camera in each of the four directions during shooting.
Referring to a flow chart of
In the flow chart of
Referring to a flow chart of
In Step S1305, the control unit 101 acquires respective maximum shake correction values in the four directions among the shake correction values acquired in Step S703. The maximum shake correction values in the fourth directions are a maximum shake correction value in a positive X-axis direction, a maximum shake correction value in a negative X-axis direction, a maximum shake correction value in a positive Y-axis direction, and a maximum shake correction value in a negative Y-axis direction. The maximum shake correction values in the four directions are values according to magnitudes of camera shakes in the four directions resulting from camera shake.
In Step S1306, the control unit 101 determines the invalid regions according to the maximum shake correction values in the individual directions, which have been determined in Step S1305. This can be achieved by, e.g., preparing mask images for the invalid regions in several steps corresponding to the shake correction values in the individual directions in advance and choosing the mask images for the invalid regions according to the shake correction values equal to or more than the maximum shake correction values in the individual directions.
For example,
Thus, in the mask image according to the shake correction value in the positive X-axis direction, the region representing the invalid region appears on a right side of the image. Meanwhile, in the mask image according to the shake correction value in the negative X-axis direction, the region representing the invalid region appears on a left side of the image.
Likewise, in the mask image according to the shake correction value in the positive Y-axis direction, the region representing the invalid region appears below the image while, in the mask image according to the shake correction value in the negative Y-axis direction, the region representing the invalid region appears above the image.
In Step S1307, the control unit 101 acquires a logical OR of the mask images for the invalid regions in the four directions, which have been determined in Step S1306, as an invalid region common to the VR video (all the frames).
For example, a case is assumed in which the mask image when the shake correction value in the positive X-axis direction is small, the mask image when the shake correction value in the negative X-axis direction is large, the mask image when the shake correction value in the positive Y-axis direction is intermediate, and the mask image when the shake correction value in the negative Y-axis direction is 0 are selected. In this case, the control unit 101 acquires such a mask image 1500 as to represent the invalid regions (regions represented in black) corresponding to the logical OR of the four mask images 1500, as illustrated in
According to the third embodiment, there is no loop processing for each of the pixels in Steps S705 to S710 in the flow chart of
Information on the invalid regions in the equirectangular images previously calculated can be used again in a case where a specified condition is satisfied. Examples of the case where the specified condition is satisfied include a case where the method of calculating the invalid region is the same and the manual correction parameters are also the same.
Referring to a flow chart of
In Step S1603, the control unit 101 determines whether or not invalid region information (information including the invalid region, setting for calculating the invalid region, and the like) of the equirectangular image is stored in a file. When it is determined that the invalid region information is stored in the file (i.e., the information related to the invalid region calculated during previous image conversion is stored in the file), a flow advances to Step S1604. When it is determined that the invalid region information is not stored in the file, the flow advances to Step S503. The file in which the invalid region information of the equirectangular image is stored herein may be the video file, or may be a file other than the video file.
In Step S1604, the control unit 101 determines whether or not the invalid region of the equirectangular image stored in the file has been calculated on the basis of the same setting as the current setting. Examples of the “setting” mentioned herein include a method of calculating the invalid region or the manual correction parameter values. When it is determined that the invalid region has been calculated on the basis of the same setting as the current setting, a flow advances to Step S1605. When it is determined that the invalid region has not been calculated on the basis of the same setting as the current setting, the flow advances to Step S503.
In Step S1605, the control unit 101 reads the invalid region information of the equirectangular image stored in the file. By the processing, the equirectangular image (VR video) is generated on the basis of the invalid region information in Step S504 subsequent thereto.
In Step S1607, the control unit 101 stores, in the file, the invalid region of the equirectangular image in the video file calculated by Step S503, the setting for calculating the invalid region, the manual correction parameters, and the like as the invalid region information. It is to be noted herein that the file in which the invalid region information is to be stored may be the video file or the file other than the video file.
According to the fourth embodiment, when the appropriate invalid region information is stored in advance, the processing of calculating the invalid region can be omitted, and therefore the equirectangular image can be generated at a higher speed.
A plurality of the video files may be continuously reproduced as one video. Accordingly, in a fifth embodiment, the editing apparatus 100 performs control such that the positions and shapes of the invalid regions do not vary not on a per video-file basis, but on a per reproduction basis (throughout the plurality of video files).
Referring to a flow chart of
Note that, in Step S503, with regard to all the video files to be continuously reproduced (all the video files that can continuously be reproduced), the same invalid coordinate table is used. As a result, in the invalid coordinate table, coordinates of a pixel determined to be the invalid pixel in any of frames of any of the video files are registered. In Step S504 also, using the same invalid coordinate table for each of the video files, the equirectangular image is output (generated).
In Step S1701, the control unit 101 sets the variable n representing the file number of the video file to 0.
In Step S1702, the control unit 101 determines, for each of all the video files, whether or not the processing in Steps S501 to S503 has been performed. When it is determined that the processing in Steps S501 to S503 has been performed for each of all the video files, a flow advances to Step S1707. When it is determined that the processing in Steps S501 to S503 has not been performed for at least any of all the video files, the flow advances to Step S501.
In Step S1706, the control unit 101 adds 1 to the variable n representing the file number.
In Step S1707, the control unit 101 sets the variable n representing the file number to 0 again.
In Step S1708, the control unit 101 determines whether or not processing in Step S504 has been performed for each of all the video files. When it is determined that the processing in Step S504 has been performed for each of all the video files, the processing in the present flow chart is completed. When it is determined that the processing in Step S504 has not been performed for any of all the video files, the processing in the present flow chart advances to Step S504.
In Step S1710, the control unit 101 adds 1 to the variable n representing the file number.
In the fifth embodiment, it is possible to provide each of the invalid regions formed in the equirectangular images with the same position and the same shape throughout all the frames of the plurality of video files. Thus, during reproduction of each of the equirectangular images in the plurality of video files, it is possible to suppress flickering caused by changes in the shape of the invalid region.
In a sixth embodiment, the editing apparatus 100 arithmetically determines the equirectangular image from the circular fisheye image on the basis of the image correction information such as the shake correction value (correction value for correcting camera shape). In addition, the editing apparatus 100 generates such a mask image (image mask or mask) as to partially cover the equirectangular image on the basis of the image correction information. The circular fisheye image in the sixth embodiment has two image circles (shooting regions).
In Step S501, the control unit 101 reads the common meta data stored in the file meta data region 302 of the video file through the input unit. The common meta data is meta data related to a shooting state of the video. For example, the common meta data includes the number of pixels in the video file, design values of an optical system, information on sizes of the image circles and positions thereof on a sensor, information related to a manufacturing error of the optical system, time-series information of an orientation of the optical system during video shooting, and information related to a sensor output.
In Step S502, the correction information generation unit 109 generates, e.g., the shake correction value for each of all the frames as the image correction information (geometrical image correction information) according to the flow chart of
Note that, in the sixth embodiment, in Step S609 in the flow chart of
In Step S1902, the mask generation unit 111 generates the mask image (such as a mask image 2200 as illustrated in
In Step S1903, the coordinate information generation unit 110 obtains, for each of the frames, a coordinate value of a region of the circular fisheye image corresponding to the equirectangular image to generate the equirectangular image.
In Step S1904, the coordinate information generation unit 110 applies (superimposes or combines) the mask image generated in Step S1902 to (on or with) the equirectangular image generated in Step S1903 to generate a new equirectangular image.
For example, the coordinate information generation unit 110 superimposes the mask image on the equirectangular image. At this time, the coordinate information generation unit 110 ensures that a region of the equirectangular image overlapping the masked region is hidden by the masked region and that a region of the equirectangular image overlapping the unmasked region is not hidden thereby. For example, the coordinate information generation unit 110 uses a darker pixel (with a lower brightness value) for each of the pixels in the equirectangular image which is at the same coordinates as those in the mask image to generate the new equirectangular image (equirectangular image having the mask image). Thus, the coordinate information generation unit 110 may also generate the new equirectangular image.
The sixth embodiment uses the meta data to allow a result of the camera shake correction processing to be considered for the mask image. As a result, even when a region (region outside of the image circle) unintended by a photographer is formed in the equirectangular image, it is possible to hide the region with the mask image. Consequently, a sense of immersion of the user in the VR video is improved.
Referring to a flow chart of
In Step S2001, the mask generation unit 111 sets the variable i representing the position of the pixel to 0. Hereinbelow, an i-th pixel is one pixel in the mask image. Processing including and subsequent to Step S2001 may be processed sequentially in the X-direction of the (X, Y) coordinates, or may also be processed sequentially in the Y-direction. In Step S2002, the mask generation unit 111 determines whether or not the pixel values of all the pixels in the mask image have been calculated. When it is determined that the pixel value of any of all the pixels in the mask image has not been calculated, a flow advances to Step S2003. When it is determined that the pixel values of all the pixels in the mask image have been calculated, processing in the present flow chart is ended.
In Step S2003, the mask generation unit 111 determines whether or not the i-th pixel in the mask image is included in the masked region. In the sixth embodiment, for determination of whether or not a given pixel is included in the masked region, the meta data related to the image circle 1021 and the shake correction value (geometrical image correction information) calculated in Step S502 are used.
For example, the coordinate information generation unit 110 calculates coordinates in the circular fisheye image (see
Then, the mask generation unit 111 corrects (adjusts), on the basis of the shake correction value (geometrical image correction information) calculated in Step S502, the masked region (position and size of the masked region). For example, the mask generation unit 111 refers to the shake correction value for each of the frames to accordingly widen the masked region when the camera shake correction or the like has not been performed by the shake correction value. Thus, the mask generation unit 111 calculates the masked region (position and size of the masked region) to which the camera shake correction has been added.
Referring to the masked region thus calculated, in Step S2003, the mask generation unit 111 determines whether or not the i-th pixel is included in the masked region. When it is determined that the i-th pixel is included in the masked region, the flow advances to Step S2005. When it is determined that the i-th pixel is not included in the masked region, the flow advances to Step S2004.
In Step S2004, the mask generation unit 111 sets the i-th pixel in the mask image to a white pixel (sets a pixel value indicating white as the pixel value of the i-th pixel). When being superimposed on the given region of the equirectangular image, a region of the mask image that has been set to the white pixel transmits and displays the given region (the given region is displayed without being interrupted).
In Step S2005, the mask generation unit 111 sets the i-th pixel in the mask image as a black-color pixel (sets a pixel value indicating black as the pixel value of the i-the pixel). When being superimposed on the given region of the equirectangular image, a region of the mask image that has been set to the black-color pixel gives a black color to the given region to cover the given region.
In Step S2006, the mask generation unit 111 adds 1 to the variable i. As a result, the processing in Steps S2002 to S2005 is performed on the next pixel in the mask image.
Thus, in the sixth embodiment, the pixel in the masked region is set as the black-color pixel, while a pixel in the unmasked region is set as a white-color pixel. By performing such processing, the mask generation unit 111 generates the mask image 2200, which is a distribution of white pixels and black pixels as illustrated in
The mask image 2200 illustrated in
For example, when the gradation region 2202 is superimposed on a given region of the equirectangular image, a color of the gradation region 2202 is given to the given region. Alternatively, when the gradation region 2202 is superimposed on the given region of the equirectangular image, the gradation region 2202 may also transmit the given region with a higher transmittance as the brightness value thereof is higher and display the given region. By applying the mask image having the gradation region 2202, it is possible to smoothen a brightness variation in a peripheral portion of the VR video. Therefore, it is possible to improve the sense of immersion in the VR video.
The gradation region 2202 smoothens the brightness variation at a boundary portion between the unmasked region 2201 and the masked region 2203. In the sixth embodiment, a beginning portion of the gradation region 2202 (corresponding to the boundary position between the gradation region 2202 and the unmasked region 2201) is determined on the basis of the shake correction value. The mask image having the gradation region 2202 may also be generated for each of the frames on the basis of the shake correction value for each of the frames. The beginning position of the gradation region 2202 may also be determined on the basis of a maximum value among the shake correction values for all the frames.
Referring to
A beginning position variation amount 2303, which is a difference between the gradation beginning position 2301 and the gradation beginning position 2304, corresponds to an amount of displacement of a relative position between the image circle 4011 and the region 4012, which has resulted from the camera shake correction. In other words, the gradation beginning position 2304 is shifted from the gradation beginning position 2301 in a center direction (direction in which the unmasked region 2201 is present) of the mask image 2200 by the number of pixels corresponding to the shake correction value. By thus adding information on the shake correction value with regard to the beginning position of the gradation region 2202, it is possible to smoothen the brightness variation in the peripheral portion. As a result, it is possible to more naturally hide reflection unintended by the photographer or the like with the mask image. Note that the gradation region is determined on the basis of, e.g., the white pixels and the black pixels which are set by the processing of generating the mask image (flow chart of
In the mask image 2200 illustrated in
In a case where, e.g., a fisheye lens having an angle of view over 180° is used when the video including the two image circles is to be shot, as described with reference to
Note that, when viewing the VR video, the user uses an HMD to view the video shot by the two optical systems with each of left and right eyes. At this time, when the masked region has different shapes on left and right sides, a region that can be viewed only with one of the eyes is formed, and the video may cause an uncomfortable feeling. In the sixth embodiment, to allow the masked region to have equal shapes on the left and right sides, the second masked region 2205 is configured to have symmetrical shapes in left and right videos.
Thus, in the sixth embodiment, an example using the shake correction value as the image correction information to be used to generate the mask image is shown. It is to be noted herein that, in the VR video, to allow an appropriate stereoscopic effect to be felt, it is important that a parallax between a left-eye image and a right-eye image is appropriately set. Consequently, parallax correction for adjusting positions of left and right images may be performed. In addition, when there is an unintended inclination in the VR video, the sense of immersion of the viewer may be impaired. To prevent this, horizontal correction for adjusting the inclination of the video may be performed. Therefore, the geometrical correction to be performed on the equirectangular image is not limited to the camera shake correction, and may also be either of horizontal correction and parallax correction. In addition, the image correction information may also be either of the horizontal correction and the parallax correction.
Meanwhile, with regard to the camera shake correction, the image correction information may also be calculated by using a sensor output (sensor information) from an acceleration sensor mounted in the image capturing apparatus or the like as the meta data. Alternatively, information on detected feature points in each of the frames may also be used as the meta data. Still alternatively, the meta data may also be information, which is combined information of the sensor output and the feature points. With regard to the horizontal correction, e.g., an output of a sensor included in the image capturing apparatus, such as a gyro sensor, may also be used as the meta data. With regard to the parallax correction, information related to a deviation of a real value of the optical system from a design value may also be used as the meta data. Yet alternatively, information related to relative positions of the two image circles calculated from the image may also be used as the meta data. Note that, in the sixth embodiment, the mask image is generated so as to be applied to the equirectangular image, but is not limited thereto. For example, it may also be possible to generate the mask image so as to apply the mask image to the circular fisheye image, apply the mask image to the circular fisheye image, and then obtain the equirectangular image.
In a seventh embodiment, referring to
In Step S2501, the control unit 101 generates the equirectangular image in each of all the frames, and outputs the generated equirectangular image.
In Step S2502, to the equirectangular image generated in Step S504, the mask image generated in Step S1902 is applied to generate a new equirectangular image.
It is to be noted herein that, in Step S2501 described above, the processing in the flow chart of
In Step S902 in the flow chart of
Referring to
The coordinate information generation unit 110 calculates, for a given pixel in the equirectangular image 406, the corresponding “coordinates of the pixel in the region of the circular fisheye image”. An invalid region boundary line 2601 is a line obtained by projecting a line indicating the boundary between the valid region and the invalid region in the region of the circular fisheye image on the region of the equirectangular image. In other words, in
When it is determined that the center 2603 corresponds to the invalid region of the circular fisheye image, coordinates corresponding to the center 2603 cannot be obtained. Meanwhile, “coordinates in the region of the circular fisheye image” corresponding to each of the pixels are obtained by linear interpolation using the coordinates of the two adjacent centers 2603. Accordingly, to pixels located between the “center 2603 determined to correspond to the invalid region” and the “center 2603 adjacent thereto”, the black pixels 2604 serving as the invalid pixels are input.
In
As a result, in the seventh embodiment, it is possible to hide the region of the equirectangular image resulting from an arithmetic error with the mask image, and therefore prevent the sense of immersion of the viewer when viewing the VR video from being impaired.
Note that the seventh embodiment has shown an example in which, after the equirectangular image is generated from a result of calculating the invalid region, the mask image is generated. In an order of generation, the mask image may be generated earlier, or the generation of the equirectangular image and the generation of the mask image may also be simultaneously processed in parallel.
According to the present invention, it is possible to generate a VR video with reduced flickering during reproduction.
In the foregoing, “when A is equal to or more than B, advance to Step S1 and, when A is smaller (lower) than B, advance to Step S2” may also be read as “when A is larger (higher) than B, advance to Step S1 and, when A is equal to or less than B, advance to Step S2”. Conversely, “when A is larger (higher) than B, advance to Step S1 and, when A is equal to or less than B, advance to Step S2” may also be read as “when A is equal to or more than B, advance to Step S1 and, when A is smaller (lower) than B, advance to Step S2”. Accordingly, as long as no contradiction arises, “equal to or more than A” may also be read as “larger (higher, longer, or more) than A”, while “equal to or less than A” may also be read as “smaller (lower, shorter, or less) than A”. In addition, “larger (higher, longer, or more) than A” may also be read as “equal to or more than A”, while “smaller (lower, shorter, or less) than A” may also be read as “equal to or less than A”.
Note that each of the various control described above may be or may not be performed by one hardware item (e.g., a processor or circuit). It may also be possible that a plurality of hardware items (e.g., a plurality of processors, a plurality of circuits, or a combination of one or more processors and one or more circuits) share processing and thereby control the entire apparatus.
The processor mentioned above is a processor in a broad sense of meaning, and includes a versatile processor and a dedicated processor. Examples of the versatile processor include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), and the like. Examples of the dedicated processor include a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and the like. Examples of the programmable logic device include a FPGA (Field Programmable Gate Array), a CPLD (Complex Programmable Logic Device), and the like.
While the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and encompasses various modes in a scope not departing from the gist of the invention. In addition, each of the embodiments described above merely represents an embodiment of the present invention, and the individual embodiments may also be combined with each other as appropriate.
Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2024-007917 | Jan 2024 | JP | national |
| 2024-092912 | Jun 2024 | JP | national |
