The invention relates to the technology of an unmanned aerial vehicle (UAV), and more particularly to a panoramic photographing method for an UAV and the UAV using the same.
An unmanned aerial vehicle (UAV) can provide services in many fields, such as land, surveying, forestry, transportation, water conservancy and military. A flight control device of the UAV mainly performs the flying altitude calculation of the UAV, the flying route control, the flying data feedback, and the associated flight tasks in the flying process. From the view of the structural division, the flight control system is the action center of the UAV, and the stability thereof decides the safety of the UAV in the overall flying process.
At present, the widest application of the UAV is photography. The UAV equipped with a digital camera can be used to perform aerial photography tasks. A smart mobile phone can be used to control the UAV to perform the fixed-location photography. China Patent No. CN104574278B discloses an aerial-photography image stitching method based on the local feedback mechanism and sparse global adjustment, wherein the UAV is used to perform the panoramic photography, and the images are transmitted to a terminal device through wireless streaming, and then the panoramic image stitching is performed in the terminal device.
China Patent Publication No. CN105611170A discloses an unmanned aerial vehicle and panoramic stitching method, device and system thereof, wherein the UAV is used to perform the panoramic photography, and the photographed pictures at the end of the UAV is panoramically stitched to obtain a panoramic image. In this patent, the pictures are stitched using a general scale-invariant feature transform (SIFT) operation or a speeded up robust feature (SURF) operation. China Patent Publication No. CN100517061C discloses a device for producing a panorama image and a method thereof, wherein the camera device is rotated by way of motor control to photograph and stitch the panoramic images.
In the existing patents, most panoramic images need to be positioned with the global positioning system (GPS) so that the relationship between the adjacent images can be obtained to facilitate the subsequent image stitching process. This method is only limited to the outdoor panoramic photography and the aerial view image photography, and cannot be applied to the indoor space. In the above-mentioned operation, the range of the comparing operation almost occupies 80% of the overall picture, so a lot of computation amounts need to be spent. If the processor of the digital camera equipped in the UAV is not powerful enough, then it is difficult to perform the panoramic photographing.
So, in most technology, such as CN104574278B, the photographed images are transmitted back to the terminal through the wireless transmission so that the image stitching is performed in the terminal. In CN105611170A, the image stitching is implemented in the UAV to avoid the wireless transmission and the drawback that the images need to be compressed so that the good quality can be obtained. However, this patent does not mention how to utilize the property of the UAV to reduce the calculation amount and the power consumption.
So, the reduction of the computation amount of the image stitching of the panoramic photographing is the problem needed to be urgently overcome.
An objective of the invention is to provide a panoramic photographing method for an unmanned aerial vehicle (UAV) and the UAV using the same. Upon photographing, the altitude and angle of the UAV are recorded, and the block for the image stitching operation is determined according to the altitude and angle. Thus, the computation amount of image stitching of panoramic photographing is reduced, or the panoramic picture can be directly returned on the UAV without image stitching being performed on a control terminal.
In view of this, the invention provides a panoramic photographing method for an UAV. The panoramic photographing method for the UAV includes steps of: disposing a digital camera on the UAV; recording a flying altitude and a flying angle of the UAV of each of photographed pictures when a panoramic photographing process is performed; performing an image stitching process to determine overlapped regions of the two photographed pictures according to the flying altitude and the flying angle of the UAV, and obtain a feature operation region by way of division; and performing a feature operation in the feature operation region of the two photographed pictures to determine an image stitching location, and thus generate a panoramic picture.
The invention also provides an unmanned aerial vehicle (UAV) including a flying mechanism, a wireless transceiver, a digital camera, an inertia measurement unit and a control unit. The inertia measurement unit detects a flying altitude and a flying angle, and outputs a feedback signal. The control unit is coupled to the flying mechanism, the wireless transceiver, the digital camera and the inertia measurement unit. The control unit controls the flying mechanism to control the flying altitude and the flying angle according to a signal received by the wireless transceiver, and determines the flying altitude and the flying angle according to the feedback signal of the inertia measurement unit. When the wireless transceiver receives a panoramic photographing instruction, the control unit controls the digital camera to perform photographing, and records the flying altitude and the flying angle of the UAV of each of photographed pictures. When an image stitching process is performed, overlapped regions of the two photographed pictures are determined according to the flying altitude and the flying angle, and a feature operation region is obtained by way of division, and then a feature operation is performed in the feature operation region of the two photographed pictures to determine an image stitching location, and thus generate a panoramic picture.
In the panoramic photographing method for the UAV and the UAV using the same according to the preferred embodiment of the invention, the feature operation includes a scale-invariant feature transform (SIFT) operation. In another embodiment, the feature operation includes a speeded up robust features (SURF) operation. Furthermore, in the panoramic photographing method for the UAV in a preferred embodiment, the first photographed picture is further set as a center picture of the panoramic picture.
The essence of the invention is to record the flying altitude of the UAV and the flying angle of the UAV corresponding to each photographed picture when the image stitching of panoramic photographing of the UAV is performed. Furthermore, the overlapped portion of two pictures is determined in advance according to the flying altitude of the UAV and the flying angle of the UAV in the image stitching operation. So, when the feature operation, such as the scale-invariant feature transform (SIFT) operation or the speeded up robust feature (SURF) operation, is performed, the feature operation is needed in only the overlapped regions of the two photographed pictures rather than the entire photographed picture. Thus, the panoramic photographing of the UAV can be performed with the less computation amount and the lower operation power. Furthermore, this panoramic photographing does not need the GPS, and this method can perform the photographing either indoors or outdoors.
The above-mentioned and other objects, features and advantages of the present invention will become more apparent from the following detailed descriptions of preferred embodiments thereof taken in conjunction with the accompanying drawings.
The wireless transceiver 102 in this embodiment is used for receiving a control instruction of a mobile device 106, and returning data to the mobile device 106. Although the mobile device is taken as an example in this embodiment, those skilled in the art should know that a remote controller can also be used for implementation. Therefore. the invention is not restricted thereto. The control unit 105 is coupled to the flying mechanism 101, the wireless transceiver 102, the digital camera 103 and the inertia measurement unit 104. The control unit 105 controls the flying mechanism 101 to control the flying altitude and the flying angle according to a signal received by the wireless transceiver 102, and determines the flying altitude and the flying angle according to the feedback signal FB of the inertia measurement unit 104.
In the above-mentioned image stitching process, the control unit 105 determines the overlapped regions 303 and 304 of the two photographed pictures according to the flying angle and the flying altitude recorded when the picture 301 and the picture 302 are photographed, and then predetermined ranges 305 and 306 are extended outwardly from the overlapped regions 303 and 304. The overlapped region 303 plus the predetermined range 305 is the feature operation region of the picture 301. The overlapped region 304 plus the predetermined range 306 is the feature operation region of the picture 302. Thereafter, the control unit 105 performs feature point search and feature point comparison to perform stitching of the two photographed pictures, as shown in
Compared with the image stitching process in the prior art, it is necessary to search for feature points of two photographed pictures firstly, and then compare the feature points of the two photographed pictures. However, in the prior art, when the feature points are compared, it is necessary to search for feature points of the entire photographed pictures and compare the feature points of the entire photographed pictures. Compared with the invention, the flying angle and the flying altitude of the UAV corresponding to the photographed picture are recorded firstly to obtain the overlapped regions of the two photographed pictures, so the invention can extremely narrow down the region for feature point search and comparison. That is, the invention can significantly reduce the computation amount required for the image stitching.
As can be understood from the above-mentioned description, the computation amount required for the image stitching can be significantly reduced in the invention. Therefore, the image stitching process of the invention can be directly operated in the UAV, the panoramic picture is obtained, and then the panoramic picture is outputted to the mobile device 106 through the wireless transceiver 102. In addition, when panoramic photographing is performed, the UAV returns each photographed picture and the flying altitude and the flying angle corresponding to each photographed picture to the mobile device 106, so that the above-mentioned image stitching process can also be performed in the mobile device 106. In another embodiment of the invention, when panoramic photographing is performed, the UAV can firstly determine the overlapped regions according to the flying altitude and the flying angle corresponding to each photographed picture, and then return each photographed picture and the positions of the overlapped regions of the photographed pictures to the mobile device 106, so that the mobile device 106 performs the above-mentioned image stitching process.
In the example of
In the above-mentioned panoramic photographing process, every time when a picture is photographed, the UAV rotates an angle, and a next picture is photographed again to perform the image stitching process. The magnitude of this angle of rotation is determined according to, for example, the magnitude of the lens angle of the digital camera 103 and the hardware limitation of the flying mechanism 101 and the like. For example, if the parameter of the field of view (FOV) of the lens of the digital camera 103 is larger, then it means that the lens has a larger photographing angle. Therefore, the UAV can have a larger angle of rotation every time when it photographs. In order to make the continuously photographed pictures have the overlapped regions, the angle of each rotation of the UAV must be smaller than the FOV parameters. In addition, the angle of rotation may be, for example, a fixed predetermined angle, and the predetermined angle may be preset by the system designer, may be set by the user through the mobile device 106, or may also be determined by the control unit based on the actual state of the UAV.
In the above-mentioned panoramic photographing process, every time when a picture is photographed, the control unit 105 records the feedback signal FB returned by the inertia measurement unit 104, and the control unit 105 determines the overlapped region of every picture through the feedback signal to perform the subsequent image stitching process. Generally speaking, a nine-axis or six-axis sensor is selected as the inertia measurement unit 104. Therefore, the inertia measurement unit 104 is able to return the nine-axis information or six-axis information to the control unit 105. However, in the invention embodiment, when the control unit 105 actually records and calculates the overlapped regions, only some information in the nine-axis information or six-axis information may be used. For example, the overlapped regions can be calculated using only the triaxial information. Therefore, the invention is not limited to the amount of information recorded and calculated.
In order to make those skilled in the art implement the invention according to the this embodiment, how to determine the overlapped regions of the two photographed pictures according to the flight information of the UAV in the image stitching process will be explained in the following. First, a picture is photographed at a time point T1, a photographed picture (represented as I) is generated, and the camera's coordinates at this time are WI. Here, for the sake of illustration for this embodiment, the camera's initial position coordinates are set to WI(0,0,0).
At the time point T1, the control unit will start recording the triaxial acceleration A(x,y,z) returned by the inertia measurement unit 104. Next, it is assumed that the camera is only rotated in a manner perpendicular to the Z axis. Furthermore, when the camera is rotated by an angle θ, the next photographing is performed (i.e., at the time point T2), and a photographed picture (represented as J) is generated, as shown in
In addition, the control unit records the triaxial acceleration A(x,y,z) returned by the inertia measurement unit 104 between the time points T1 and T2, so the relative displacement amount of the camera between the time points T1 and T2 is obtained through speed integration in the time domain. In other words, the control unit can obtain the camera's coordinates WJ(X,Y,Z) at the time point T2 through the speed integration and the triaxial acceleration recorded data. The camera's coordinates WI and WJ are convertible to the center coordinates CI and CJ of the lens, so that the displacement matrix may be obtained. In this embodiment, if the center of the lens at the time point T1 is set as 0, then the center of the lens at the time point T2 is offset to (cjx,cjy). According to the above-mentioned known or calculated numerical data (rotation matrix RI, rotation matrix RJ, displacement matrix Tj and focal length f), it is possible to estimate a homograph matrix, which has the mathematical equation expressed as H, and the value expressed by the following mathematical equation (1).
The digital camera photographs the photographed picture I at the time point T1 through a first focal length fI, and the control unit calculates a first focal length matrix VI according to the first focal length fI. The digital camera photographs the photographed picture J at the time point T2 through a second focal length fJ, and the control unit calculates a second focal length matrix VJ according to the second focal length fJ.
According to the mathematical equation (1), the control unit can estimate the homograph matrix, the photographed picture I and the photographed picture J can be projected onto the same plane by way of coordinate transformation. In this embodiment, if the photographed picture I is used as the reference plane, then the image obtained by projecting the photographed picture J onto the reference plane is represented as J′, and the relationship of coordinate transformation is represented by the following mathematical equations (2) and (3). In
Substituting the coordinate values of four points of the boundary of the photographed picture J into the mathematical equations (2) and (3) can obtain the projected image J′. In the above equations (2) and (3), x and y are coordinate values in the photographed picture I, respectively, x′ and y′ are coordinate values in the projected image J′. Therefore, the overlapping region in the photographed picture I can be obtained according to the above-mentioned equations (2) and (3).
As can be seen from the above-mentioned description, at least the following five steps may be summarized in the image stitching process of the embodiment of the invention.
In the first step, the control unit records the flying altitude and the flying angle of the UAV at the first time T1, and further calculates the first rotation matrix RI according to the sensor fusion algorithm and the flying altitude and the flying angle recorded at the first time. Furthermore, the control unit records the flying altitude and the flying angle of the UAV at the second time T2, and further calculates the second rotation matrix RJ according to the sensor fusion algorithm and the flying altitude and the flying angle recorded at the second time.
In the second step, a camera's focal length matrix, including the above-mentioned VI and VJ, is obtained according to a focal length of the digital camera.
In the third step, the control circuit detects the displacement of the center of the lens of the digital camera to obtain a camera displacement matrix. Those skilled in the art should know that the order of priority of the first to third steps described hereinabove are not particularly restricted.
In the fourth step, the control circuit calculates a homograph matrix according to the camera's focal length matrix, the camera displacement matrix, the first rotation matrix and the second rotation matrix.
In the five step, the image coordinates of the second photographed picture J is substituted to obtain the image coordinates corresponding to the first photographed picture I according to the homograph matrix (e.g., the above-mentioned equations (2) and (3)).
In the following, how to use the homography matrix to obtain the overlapped regions will be described.
Referring to
W
l
=x
1
−x
j0 if (x1−xj0≥x1−xj1) or
W
l
=x
1
−x
j1 if (x1−xj0<x1−xj1)
H
l
=y
j1
−y
j0 if (yj1−yj0≥y1−y0) or
H
l
=y
1
−y
0 if (y1−y0<yj1−yj0)
Referring to
W
J
=x
i0
−x
0 if (xi0−x0≥xi1−x0) or
W
J
=x
i1
−x
0 if (xi1−x0<xi0−x0)
H
l
=y
1
−y
0 if (y1−y0≥yi1−yi0) or
H
l
=y
i1
−y
i0 if (yi1−yi0<y1−y0)
In addition, for the sake of illustration of the invention, it is assumed that there are five feature points in the photographed picture I, and there are four feature points in the image of the photographed picture J. After the division of the overlapped region, there are 3 feature points of the image I falling into the overlapped region, and 3 feature points of the image J falling into the overlapped region. Therefore, the feature comparison only needs to compare the feature points falling into the region so as to save other comparisons and calculation time.
In a step S401, the method starts.
In a step S402, it is judged whether a panoramic photographing instruction is received. When the panoramic photographing instruction is received, a step S403 is performed.
In the step S403, panoramic photographing is performed to record a flying altitude and a flying angle of the UAV of each of photographed pictures. When the wireless transceiver 102 receives a panoramic photographing instruction, the panoramic photographing instruction is sent to the control unit 105. At this time, the control unit 105 controls the digital camera 103 to perform the photographing. Furthermore, when a picture is photographed at each time, the control unit 105 captures the flying attitude and flying angle of the UAV from the inertia measurement unit 104 at this time, and immediately records the flying altitude and the flying angle of the UAV when the picture is photographed.
In a step S404, an image stitching process is performed to determine overlapped regions of the two photographed pictures, and obtain a feature operation region by way of division according to the flying altitude and the flying angle of the UAV. The feature operation region is the example of
In a step S405, a feature operation is performed to determine an image stitching location in feature operation regions of the two adjacent photographed pictures, and thus to generate a panoramic picture, as shown in
In step S406, the method ends.
In addition, a step can also be inserted between the steps S405 and S406. The first photographed picture is set as a central picture to center the image of the first viewing angle of the first picture to facilitate the picture preview for the user.
In summary, the essence of the invention is to record the flying altitude of the UAV and the flying angle of the UAV corresponding to each photographed picture when the image stitching of panoramic photographing of the UAV is performed. Furthermore, the overlapped portion of two pictures is determined in advance according to the flying altitude of the UAV and the flying angle of the UAV in the image stitching operation. So, when the feature operation is performed, the feature operation is needed in only the overlapped regions of the two photographed pictures rather than the entire photographed picture. Thus, the panoramic photographing of the UAV can be performed with the less computation amount and the lower operation power. Furthermore, this panoramic photographing does not need the GPS, and this method can perform the photographing either indoors or outdoors.
While the present invention has been described by way of examples and in terms of preferred embodiments, it is to be understood that the present invention is not limited thereto. To the contrary, it is intended to cover various modifications. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications.
Number | Date | Country | Kind |
---|---|---|---|
106142798 | Dec 2017 | TW | national |