Panoramic video anti-shake method and portable terminal

Abstract

The present invention is applicable to the field of videos. Provided are a panoramic video anti-shake method and a portable terminal. The method comprises: acquiring world coordinates of any reference point in a world coordinate system in real time, and also acquiring the corresponding camera coordinates of the reference point in the portable terminal and an angular velocity value of a gyroscope at a current state in the portable terminal; smoothing the motion of a camera by using an extended Kalman filter; decomposing the smoothed motion, synthesizing the motion into a virtual lens motion in an ePTZ mode, and calculating a rotation amount of the virtual lens; and re-projecting an original video according to the rotation amount of the virtual lens and a rotation matrix for converting from the camera coordinates to the world coordinates, so as to generate a stable video.

Description

CROSS REFERENCE TO RELATED APPLICATION

The present application is a National Phase of International Application of PCT/CN2019/109483 filed on Sep. 30, 2019 which claims priority to Chinese Patent Application No. 201811549643.6, filed on Dec. 18, 2018 and entitled “Panoramic Video Anti-Shake Method and Portable Terminal”, and the content of which is herein incorporated by reference in their entireties.

TECHNICAL FIELD

The present disclosure relates to the field of panoramic video, and particularly to a panoramic video anti-shake method and a portable terminal.

BACKGROUND

At present, when a panoramic video is shot, it is usually to hold a panoramic shooting device in hand to shoot. When a mobile shooting is taken, a phenomenon of jitter occurs in the panoramic shooting video due to unstable hands. When a panoramic view angle is acquired, the focus of the original lens is often lost due to the movement or shaking of the camera, which affects the viewing experience of the panoramic video. One of the current solutions is to use a pan-tilt to stabilize the panoramic shooting device to stabilize the pictures taken. However, the disadvantage is that the pan-tilt is more expensive and the volume is generally larger, and the pan-tilt does not completely solve the problem of picture jitter when shooting the video with a handheld panoramic shooting device.

When a panoramic video viewer is watching the panoramic video and only wants to view from angles of view in the vertical and horizontal directions of the panoramic video, it is necessary to keep the angle of view of the video changing in the vertical and horizontal directions and keep the video stable, thus it is necessary to study a panoramic video anti-shake method which can only retains the motion states in the vertical and horizontal directions.

SUMMARY

Technical Problem

The purpose of the present disclosure is to provide a panoramic video anti-shake method, a computer-readable storage medium and a portable terminal, and is intended to solve the problem of the loss of the original lens focus when the panoramic video perspective is acquired. The method can generate smooth and stable videos in the vertical and horizontal directions, and retain the original shooting angle of the camera.

Technical Solution

In the first aspect, the present disclosure provides a panoramic video anti-shake method, which includes:

acquiring a world coordinate of any one reference point in a world coordinate system in real time, and simultaneously acquiring a camera coordinate corresponding to the reference point in a portable terminal and an angular velocity value of a gyroscope in the portable terminal in a current state;

smoothing a motion of the camera by using an extended Kalman filter;

decomposing the smoothed motion, synthesizing a motion of a virtual lens in an ePTZ mode, and calculating a rotation quantity of the virtual lens;

re-projecting an original video according to the rotation quantity of the virtual lens and a rotation matrix by which the camera coordinate is transformed to the world coordinate, to generate a stable video.

In the second aspect, the present disclosure provides a computer-readable storage medium, on which a computer program is stored, the computer program, when executed by a processor, implements the steps of the above-mentioned panoramic video anti-shake method.

In the third aspect, the present disclosure provides a portable terminal, which includes:

one or more processors;

a memory; and

one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, wherein the processor, when executing the computer programs, implement the steps of the above-mentioned panoramic video anti-shake method.

Advantages

In the present disclosure, by decomposing the motion of the camera and synthesizing the motion of the virtual lens in the ePTZ mode in which only the vertical and horizontal motions are retained, the focus of the lens can keep changing in the vertical and/or horizontal direction during play of the panoramic video, while the changes in other directions of the motion of the original camera are filtered out. Therefore, this method can maintain smooth motion of the rendering lens, generate a stable video and retain the original shooting angle of the camera, which has strong robustness to large noise scenes and most sports scenes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart showing a panoramic video anti-shake method according to an embodiment I of the present disclosure.

FIG. 2 is a schematic structure diagram of a portable terminal according to an embodiment III of the present disclosure.

DETAILED DESCRIPTION

In order to make the objectives, technical solution, and advantages of the present disclosure clearer, the present disclosure will be described in detail with reference to the accompanying drawings and embodiments. It should be appreciated that the specific embodiments described here are only used for explaining the present disclosure, rather than limiting the present disclosure.

In order to illustrate the technical solution of the present disclosure, specific embodiments are used for description below.

Embodiment I

Referring to FIG. 1, the panoramic video anti-shake method provided by the Embodiment 1 of the present disclosure includes the following steps.

S101: a world coordinate of any one reference point in a world coordinate system is acquired in real time, and a camera coordinate corresponding to the reference point in a portable terminal and an angular velocity value of a gyroscope in the portable terminal in a current state are simultaneously acquired.

In the embodiment I of the present disclosure, S101 can specifically be as follows.

The world coordinate of the reference point is P_w, and the camera coordinate is P_c, the following relation is specifically included:P_w=R_w2cP_c  (1)

In the formula (1),

$R_{w2c}=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]$ is a rotation matrix by which the camera coordinate is transformed to the world coordinate; the elements r₁₁ to r₃₃ are elements of the rotation matrix R_w2c,

${\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]}^T·\hspace{1em}\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]·{\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]}^T=I,$ I is the unit matrix.

The step of acquiring the angular velocity value of the gyroscope in the portable terminal in real time specifically includes: an angular velocity sensor is adopted to read a three-axis angular velocity value as w_k.

S102: an extended Kalman filter is utilized to smooth a motion of the camera.

The extended Kalman filter algorithm linearizes the nonlinear system, and then performs Kalman filter. The Kalman filter is a high-efficiency recursive filter which can estimate a state of a dynamic system from a series of measurements that do not completely contain noises.

In the embodiment I of the present disclosure, S102 can specifically be as follows.

The extended Kalman filter algorithm is utilized to establish a state model and an observation model of a motion state of the camera; specifically:

the state model is:

$\begin{array}{cc}\{\begin{array}{c}{\tilde{q}}_k=\Phi \left({\tilde{w}}_{k-1}\right)·{\tilde{q}}_{k-1}\\ {\tilde{w}}_k={\tilde{w}}_{k-1}\end{array};& \left(2\right)\end{array}$

the observation model is:

$\begin{array}{cc}\{\begin{array}{c}{q}_k={\tilde{q}}_k\\ w_k={\tilde{w}}_k\end{array}.& \left(3\right)\end{array}$

In the formulas (2) and (3), k is time, w_k is an obtained angular velocity, and q_k is an obtained observation vector of the rotation quantity; {tilde over (w)}_k and {tilde over (q)}_k are state values of the angular velocity and rotation quantity, {tilde over (w)}_k−1 are {tilde over (q)}_k−1 state values of the angular velocity and rotation quantity at time k−1; q_k is a quaternion representation of R_w2c⁻¹; w_k is the angular velocity value of the gyroscope; w_k=lg(q_k+1·q_k⁻¹, Φ({tilde over (w)}_k−1) is a state transition matrix at the time k−1; Φ({tilde over (w)}_k−1)=exp([{tilde over (w)}_k−1]_x), {tilde over (q)}_k is the quaternion representation of the estimated smoothed motion of the camera, {tilde over (q)}_k is the state value estimated by the value of {tilde over (q)}_k−1 at the previous time.

The specific process of updating the prediction includes: at time k, {tilde over (q)}_k−1 estimated at the previous time and the observation value {tilde over (q)}_k the current time are utilized to update the estimation of the state variable {tilde over (q)}_k to obtain an estimated value at the current time. The predicted value {tilde over (q)}_k is the rotation quantity of the virtual lens at the k-th time.

S103: the smoothed motion is decomposed, a motion of the virtual lens is synthesized in an ePTZ mode, and the rotation quantity of the virtual lens is calculated.

In the embodiment I of the present disclosure, S103 can specifically be as follows.

The coordinate of the reference point in the virtual lens is P_c, the following relation is specifically included:P_c=R_c2wP_w;  (4)

In formula (4), R_c2w is a 3*3 matrix, which is the rotation quantity of the virtual lens;

the ePTZ mode is a mode in which only the motions in the vertical and horizontal directions are retained; for the virtual lens in the synthesized ePTZ mode, the rotation quantity is set as R_c2w=Φ({tilde over (γ)}, {tilde over (φ)}), Φ({tilde over (γ)}, {tilde over (φ)})=exp({tilde over (γ)}, {tilde over (φ)}), {tilde over (γ)} is the smoothed motion trajectory of the camera in the vertical direction, and {tilde over (φ)} is the smoothed motion trajectory of the camera in the horizontal direction;

the step of decomposing the smoothed motion, synthesizing the motion of the virtual lens in the ePTZ mode and calculating the rotation quantity of the virtual lens specifically includes:

a focus direction {right arrow over (x)} of the original lens is given, the smoothed viewpoint direction is {right arrow over ({tilde over (x)})}=q_k⊗{right arrow over (x)}.

R_c2w=[{right arrow over (e)}₁, {right arrow over (e)}₂, {right arrow over (e)}₃], where

${\stackrel{\rightharpoonup }{e}}_1=\frac{\stackrel{\rightharpoonup }{x}}{\left|\stackrel{\rightharpoonup }{x}\right|},$ {right arrow over (e)}₃={right arrow over (e)}₁×{right arrow over (e)}_up, {right arrow over (e)}₂={right arrow over (e)}₃×{right arrow over (e)}₁, ⊗ represents the vector rotation in the quaternion space, and {right arrow over (e)}_up is an upward direction of the virtual lens and can be set as [0,0,1]^T.

It should be noted that the Rodrigues formula can be utilized to obtain the rotation matrix R from the quaternion after the unit vector is rotated by an angle θ. Specifically, the quaternion is set as q=(θ,x,y,z)^T, then calculation formula of the rotation matrix R is:

$R=\left[\begin{array}{ccc}\begin{array}{c}\cos \theta +\\ x^2\left(1-\cos \theta \right)\end{array}& \begin{array}{c}-zs\mathrm{in}\theta +\\ \mathrm{xy}\left(1-\cos \theta \right)\end{array}& \begin{array}{c}y\sin \theta +\\ \mathrm{xz}\left(1-\cos \theta \right)\end{array}\\ \begin{array}{c}z\sin \theta +\\ \mathrm{xy}\left(1-\cos \theta \right)\end{array}& \begin{array}{c}\cos \theta +\\ y^2\left(1-\cos \theta \right)\end{array}& \begin{array}{c}-x\sin \theta +\\ \mathrm{yz}\left(1-\cos \theta \right)\end{array}\\ \begin{array}{c}-y\sin \theta +\\ \mathrm{xz}\left(1-\cos \theta \right)\end{array}& \begin{array}{c}x\sin \theta +\\ \mathrm{yz}\left(1-\cos \theta \right)\end{array}& \begin{array}{c}\cos \theta +\\ z^2\left(1-\cos \theta \right)\end{array}\end{array}\right].$

S104: the original video is re-projected according to the rotation quantity of the virtual lens and the rotation matrix by which the camera coordinate is transformed to the world coordinate, to generate a stable video.

In the embodiment I of the present disclosure, S104 can specifically be as follows.

A corresponding relationship between a pixel in the original video frame and a pixel in an output video frame is calculated, and then interpolation resampling is performed on the original video frame according to the corresponding relationship to generate the output video frame, and finally a stable video is generated.

The pixel in the original video frame is set as P_s, and the pixel in the corresponding output video frame is P_d, then the corresponding relationship is: P_s=D_c(K_cR_w2c⁻¹R_c2w⁻¹K_c⁻¹P_d), where P_s=[x_s,y_s]^T, P_d[x_d,y_d]^T, x_s and y_s are the coordinate values of the abscissa and ordinate of the pixel P_s in the original video frame; x_d and y_d are the coordinate values of the abscissa and ordinate of the pixel P_d in the output video frame respectively; K_c and D_c are respectively an internal parameter and a distortion model of the camera, K_cis a projection internal parameter of the virtual lens.

Then the step of performing the interpolation resampling on the original video frame I_s according to the corresponding relationship and generating the output video frame I_d specifically includes:

$\begin{array}{cc}{I}_d\left(P_d\right)=\frac{\sum w_i{I}_s\left(P_s^i\right)}{\sum w_i}& \left(5\right)\end{array}$

In formula (5), w_i is the interpolation weight, and P_sⁱϵU(P_s,δ) is a neighborhood coordinate of P_s.

Embodiment II

In the embodiment II of the present disclosure, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the panoramic video anti-shake method provided in the embodiment I of the present disclosure.

The computer-readable storage medium can be a non-transitory computer-readable storage medium.

Embodiment III

FIG. 2 shows a specific structure block diagram of a portable terminal provided in the Embodiment III of the present disclosure. The portable terminal 100 includes: one or more processors 101, a memory 102, and one or more computer programs; the processor 101 is connected to the memory 102 by a bus; the one or more computer programs are stored in the memory 102, and are configured to be executed by the one or more processors 101; and the processor 101, when executing the computer program, implements the steps of the panoramic video anti-shake method provided in the embodiment I of the present disclosure.

In the embodiment of the present disclosure, by decomposing the motion of the camera and synthesizing the motion of the virtual lens in the ePTZ mode in which only the vertical and horizontal motions are retained, the focus of the lens can keep changing in the vertical direction and/or horizontal direction when the user is watching a panoramic video. Changes of the motion of the original lens in other directions are filtered out to only retain the motion of the panoramic video in the horizontal direction, accordingly this method can maintain the smooth motion of the rendering lens, generate a stable video, and retain the original shooting perspective of the camera, which has strong robustness to large noise scenes and most sports scenes.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and the storage medium can include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

The above are merely the preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modification, equivalent replacement and improvement made within the spirit and principle of the present disclosure shall be regarded as the protection scope of the present disclosure.

Claims

1. A panoramic video anti-shake method, comprising: acquiring a world coordinate of any one reference point in a world coordinate system in real time, and simultaneously acquiring a camera coordinate corresponding to the reference point in a portable terminal and an angular velocity value of a gyroscope in the portable terminal in a current state;smoothing a motion of the camera by using an extended Kalman filter;decomposing the smoothed motion, synthesizing a motion of a virtual lens in an ePTZ mode, and calculating a rotation quantity of the virtual lens;re-projecting an original video according to the rotation quantity of the virtual lens and a rotation matrix by which the camera coordinate is transformed to the world coordinate, to generate a stable video;wherein the smoothing the motion of the camera by using the extended Kalman filter specifically comprises:using a state model

2. The method according to claim 1, wherein the world coordinate of the reference point is Pw, the camera coordinate is Pc, and an angular velocity value is wk, the following relation is specifically comprised: Pw=Rw2cPc wherein,

3. The method according to claim 1, wherein a coordinate of the reference point in the virtual lens is Pc, the following relation is specifically comprised: Pc=Rc2wPw wherein Rc2w is a 3*3 matrix and is the rotation quantity of the virtual lens.

4. The method according to claim 1, wherein the re-projecting the original video according to the rotation quantity of the virtual lens and the rotation matrix by which the camera coordinate is transformed to the world coordinate to generate the stable video specifically comprises: calculating a corresponding relationship between a pixel in an original video frame and a pixel in an output video frame, and then performing interpolation sampling on the original video frame according to the corresponding relationship, generating the output video frame and finally generating the stable video;wherein, the pixel in the original video frame is set as Ps, and the pixel in the corresponding output video frame is Pd, accordingly the corresponding relationship is:

5. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the panoramic video anti-shake method according to claim 1.

6. A portable terminal, comprising: one or more processors;a memory; andone or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, wherein the processor, when executing the computer programs, implement following steps of a panoramic video anti-shake method:acquiring a world coordinate of any one reference point in a world coordinate system in real time, and simultaneously acquiring a camera coordinate corresponding to the reference point in a portable terminal and an angular velocity value of a gyroscope in the portable terminal in a current state;smoothing a motion of the camera by using an extended Kalman filter;decomposing the smoothed motion, synthesizing a motion of a virtual lens in an ePTZ mode, and calculating a rotation quantity of the virtual lens;re-projecting an original video according to the rotation quantity of the virtual lens and a rotation matrix by which the camera coordinate is transformed to the world coordinate, to generate a stable video;wherein the smoothing the motion of the camera by using the extended Kalman filter specifically comprises:using a state model

7. The portable terminal according to claim 6, wherein the world coordinate of the reference point is Pw, the camera coordinate is Pc, and an angular velocity value is wk, the method specifically comprises: Pw=Rw2cPc wherein,

8. The portable terminal according to claim 6, wherein a coordinate of the reference point in the virtual lens is Pc, the following relation is specifically comprised: Pc=Rc2wPw wherein Rc2w is a 3*3 matrix and is the rotation quantity of the virtual lens.

9. The portable terminal according to claim 6, wherein the re-projecting the original video according to the rotation quantity of the virtual lens and the rotation matrix by which the camera coordinate is transformed to the world coordinate to generate the stable video specifically comprises: calculating a corresponding relationship between a pixel in an original video frame and a pixel in an output video frame, and then performing interpolation sampling on the original video frame according to the corresponding relationship, generating the output video frame and finally generating the stable video;wherein, the pixel in the original video frame is set as Ps, and the pixel in the corresponding output video frame is Pd, accordingly the corresponding relationship is: