UNMANNED PARKING SPACE DETECTION METHOD BASED ON PANORAMIC SURROUND VIEW

Abstract

An unmanned parking space detection method based on a panoramic surround view is provided. The unmanned parking space detection method adopts a Laplace fusion algorithm to splice images to obtain the panoramic surround view, which effectively realize fusion of the images. By using an image quality evaluation function, complexity of the unmanned parking space detection method is reduced while ensuring a quality of the splicing. The panoramic surround view generated by the present disclosure has a more natural transition in the areas being spliced and fused, and the panoramic surround view display is more complete and beautiful. Compared with conventional machine vision detection technologies, the unmanned parking space detection method adopts a target detection technology for detecting parking space entrance lines and adopts a more lightweight network design to ensure real-time performance of a detection algorithm thereof, which has higher detection accuracy and stronger robustness.

Description

TECHNICAL FIELD

The present disclosure relates to a technical field of intelligent vehicles, and in particular to an unmanned parking space detection method based on a panoramic surround view.

BACKGROUND

With development of automotive technology, more and more intelligent vehicles are equipped with advanced driver assistance systems. As one of core technologies of assisted driving, perception technology has become an important way for transition of the intelligent vehicles from assisted driving to unmanned driving.

Currently, gradual increase in the number of urban vehicles has brought challenges to urban transportation, making an urban traffic situation more and more harsh and complex. Since the number of urban parking spaces is difficult to keep up with a growth in the number of automobiles, tension in parking space resources is exacerbated. Parking space detection technology is a key technology used in an unmanned parking environment sensing system. An effect of parking space detection greatly affects safety and stability of a parking process. Through the parking space detection technology, a driverless vehicle is able to obtain information of surrounding parking spaces in real time according to parking space marking lines on a parking lot and accurately judge whether a parking space thereof is valid or not.

In prior art, the driverless vehicle mainly obtains information about a surrounding environment through a certain number of cameras and obtains parking space information based on machine vision detection technology. However, a method of fusing images captured by the cameras increases cost, the images are unable to be effectively fused in the prior art, and detection accuracy of an algorithm thereof is poor.

SUMMARY

An object of the present disclosure is to provide an unmanned parking space detection method based on a panoramic surround view to solve technical problems of high cost, ineffective fusion of images, and poor detection accuracy of the prior art.

The unmanned parking space detection method based on the panoramic surround view comprises:

• • a step 1: respectively acquiring a front original image, a rear original image, a left original image, and a right original image of a surrounding environment of an automobile through a front-view camera, a rear-view camera, a left-view camera and a right-view camera arranged on the automobile;
  • a step 2: respectively performing distortion removal processing and perspective transformation processing on the front original image, the rear original image, the left original image, and the right original image to obtain a front view, a rear view, a left view and a right view;
  • a step 3: extracting feature points from the front view, the rear view, the left view and the right view by a random sample consensus (RANSAC) algorithm, and determining areas to be spliced and fused;
  • a step 4: fusing the areas to be spliced and fused by a Laplace fusion algorithm to obtain the panoramic surround view, and in the fusion process, adjusting a quantity of layers of a Laplace pyramid according to an image quality evaluation function; and a step 5: inputting the panoramic surround view into a trained parking space detection model based on deep learning to obtain a parking space attribute result and a parking space identification point result.

The trained parking space detection model based on deep learning uses a repeated visual geometry group network (REP-VGG) network as a network trunk, and after the network trunk is output, the network trunk passes through a Sim-CSPSPPF module and residual convolution modules, branches of the network trunk pass through the residual convolution modules, and different parking space results are output on feature maps with different resolutions. The parking space attribute result is predicted by a first feature map with a lower resolution, and the parking space identification point result is predicted by a second feature map with a higher resolution.

The unmanned parking space detection method based on the panoramic surround view of the present disclosure has following characteristics.

The unmanned parking space detection method provided by the present disclosure only requires four cameras to complete tasks of image acquisition and splicing, and cost thereof is inexpensive.

The present disclosure adopts the Laplace fusion algorithm to splice the front view, the rear view, the left view, and the right view to obtain the panoramic surround view, which effectively realize fusion of the front view, the rear view, the left view, and the right view. By using the image quality evaluation function, complexity of the unmanned parking space detection method is reduced while ensuring a quality of the splicing. The panoramic surround view generated by the present disclosure has a more natural transition in the areas being spliced and fused, and the panoramic surround view display is more complete and beautiful.

Compared with conventional machine vision detection technologies, the unmanned parking space detection method based on the panoramic surround view of the present disclosure adopts a target detection technology for detecting parking space entrance lines, which has higher detection accuracy and stronger robustness. The present disclosure adopts a more lightweight network design to ensure real-time performance of a detection algorithm thereof.

The unmanned parking space detection method based on the panoramic surround view of the present disclosure does not rely on relative spatial locations of vehicles around a target parking space, but only requires a presence of the parking entrance lines on the ground in a parking area that meets parking standards to accomplish a parking space detection task.

The unmanned parking space detection method based on the panoramic surround view of the present disclosure is applicable to a wide range of parking environments, and is able to detect whether the target parking space is a vertical parking space, a parallel parking space, or an inclined parking space, and is able to obtain information about whether the target parking space is occupied.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flow chart of an unmanned parking space detection method based on a panoramic surround view of the present disclosure.

FIG. 2 is a schematic diagram showing mounting positions of cameras of the present disclosure.

DETAILED DESCRIPTION

In order to make objectives, technical solutions, and advantages of the embodiments of the present disclosure clear, technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present disclosure.

As shown in FIG. 1, an unmanned parking space detection method based on a panoramic surround view of the present disclosure comprises steps 1-5.

The step 1 comprises respectively acquiring a front original image, a rear original image, a left original image, and a right original image of a surrounding environment of an automobile through a front-view camera, a rear-view camera, a left-view camera and a right-view camera arranged on the automobile.

In the embodiment, mounting positions of the front-view camera, the rear-view camera, the left-view camera, and the right-view camera are shown in FIG. 2, and the four cameras are fisheye cameras with a viewing angle of 190°, which enable a sensing range thereof to completely cover the surrounding environment of the automobile.

The step 2 comprises respectively performing distortion removal processing and perspective transformation processing on the front original image, the rear original image, the left original image, and the right original image to obtain a front view, a rear view, a left view and a right view.

In the embodiment, only radial distortions and tangential distortions of the fisheye cameras are considered.

The step 2 comprises steps 201-205.

The step 201 comprises converting physical coordinates in a world coordinate system to pixel coordinates in a pixel coordinate system by following formula:

$Z\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)=D\left(\begin{array}{ccc}{f}_x& 0& C_x\\ 0& f_y& C_y\\ 0& 0& 1\end{array}\right)\left(\begin{array}{cc}R& T\\ 0& 1\end{array}\right)\left(\begin{array}{c}U\\ V\\ W\\ 1\end{array}\right)=\mathrm{DAE}\left(\begin{array}{c}U\\ V\\ W\\ 1\end{array}\right).$

Z is a scale factor, D is a distortion matrix, R is a rotation matrix, T is a translation vector, A is an intrinsic parameter matrix,

$A=\left(\begin{array}{ccc}{f}_x& 0& C_x\\ 0& f_y& C_y\\ 0& 0& 1\end{array}\right),$

E is an extrinsic parameter matrix,

$E=\left(\begin{array}{cc}R& T\\ 0& 1\end{array}\right),\left(U,V,W\right)$

is physical coordinates of a target point in the world coordinate system, (u, v) is pixel coordinates of the target point in the pixel coordinate system, the pixel coordinate system is ideal and undistorted, f_x is a normalized focal length on an x-axis, f_y is a normalized focal length on a y-axis, f_x=f/dx, f_y=f/dy, f represents a focal length of a corresponding camera, dx and dy are pixel units, and (C_x, C_y) is an origin of a corresponding image.

The step 202 comprises fixing the world coordinate system on a checkerboard, so physical coordinates of any point W on the checkerboard is 0, and obtaining a plurality of (u, v), (U, V) by calibrating to obtain parameters of the intrinsic parameter matrix A.

The step 203 comprises obtaining an ideal matrix form of a conversion relationship between image coordinates and the pixel coordinates based on radial distortion formulas and tangential distortion formulas:

$D_1\left[\begin{array}{c}{k}_1\\ k_2\\ k_3\end{array}\right]D_2\left[\begin{array}{c}{p}_1\\ p_2\end{array}\right]=D\left[\begin{array}{c}{k}_1\\ k_2\\ k_3\end{array}\right]\left[\begin{array}{c}{p}_1\\ p_2\end{array}\right]=\left[\begin{array}{c}\hat{u}-u\\ \hat{v}-v\end{array}\right].$

The radial distortion formulas are:

$\hat{x}=x\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right),\mathrm{and}\hat{y}=y\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right).$

The tangential distortion formulas are:

$\hat{x}=x+\left(2p_{1}y+p_2\left(r^2+2x^2\right)\right),\mathrm{and}\hat{y}=y+\left(p_1\left(r^2+2y^2\right)+2p_{2}x\right).$

D₁ and D₂ are intermediate values of the distortion matrix, (x, y) are ideal undistorted normalized image coordinates, ({circumflex over (x)},ŷ) are distorted normalized image coordinates, r is a distance from an image pixel to an image center, r=√{square root over (x²+y²)}, k₁, k₂, k₃, p₁, p₂ are distortion parameters, (û, {circumflex over (v)}) are distorted pixel coordinates, and the conversion relationship between the image coordinates and the pixel coordinates is

$\hat{u}=\frac{\hat{x}}{dx}+C_x\mathrm{and}\hat{v}=\frac{\hat{y}}{\mathrm{dy}}+C_y.$

The step 204 comprises determining D₁ and D₂ by obtaining the distortion parameters through offline calibration images, so as to determine the distortion matrix D, and respectively performing the distortion removal processing on the front original image, the rear original image, the left original image, and the right original image according to the distortion matrix to obtain a dedistorted front original image, a dedistorted rear original image, a dedistorted left original image, and a dedistorted right original image.

The step 205 comprises respectively performing the perspective transformation processing on the dedistorted front original image, the dedistorted rear original image, the dedistorted left original image, and the dedistorted right original image according to the extrinsic parameter matrix E obtained by extrinsic parameter calibration to obtain the front view, the rear view, the left view, and the right view.

As shown in FIG. 2, the step 2 is performed to obtain the front view A, the right view B, the rear view C, and the left view D.

The step 3 comprises extracting feature points from the front view, the rear view, the left view, and the right view by a random sample consensus (RANSAC) algorithm, and determining areas to be spliced and fused.

As shown in FIG. 2, a feature-based image splicing method is configured for the areas that need image registration, and the RANSAC algorithm is configured to extract the feature points to determine the areas A-B, B-A, B-C, C-B, C-D, D-C, D-A, and A-D that need to be spliced and fused.

The step 4 comprises fusing the areas to be spliced and fused by a Laplace fusion algorithm to obtain the panoramic surround view, and in the fusion process, adjusting a quantity of layers of a Laplace pyramid according to an image quality evaluation function.

In the step 4, fusing the areas to be spliced and fused by the Laplace fusion algorithm to obtain the panoramic surround view comprises steps 401a-406a.

The step 401a comprises performing Gaussian smoothing and down-sampling on a first image P1 in two images to be spliced and fused and performing Gaussian smoothing and up-sampling on the first image P1, and respectively repeating for N times to obtain an (N+1)-layer Gaussian pyramid of the first image P1, obtaining a first intermediate image P_down_iof an i-th layer of the (N+1)-layer Gaussian pyramid during an i-th Gaussian smoothing and down-sampling, obtaining a second intermediate image P_up_i of the i-th layer of the (N+1)-layer Gaussian pyramid during an i-th Gaussian smoothing and up-sampling.

i≤N and the (N+1)-layer Gaussian pyramid of the first image P1 comprises N second intermediate images P_up_i and the first image PT.

The step 402a comprises constructing the Laplace pyramid of the first image P1 by following formula:

${\mathrm{P1\_L}}_i={\mathrm{P\_down}}_i-\mathrm{UP}\left({\mathrm{P\_down}}_{i+1}\right).$

P1_L_i represents a Laplacian residual image of an i-th layer of the Laplace pyramid of the first image P1, UP represent an up-sampling operation, N Laplacian residual images P1_L_i constitute the Laplacian pyramid having N layers of the first image P1, and a third intermediate image P1_L_top of the first image P1 is obtained by performing the Gaussian smoothing and down-sampling on a Laplacian residual image P1_L_N of an N-th layer of the Laplace pyramid of the first image PT.

The step 403a comprises processing a second image P2 in the two images to be spliced and fused according to the steps 401a-402a to obtain a Laplace residual image P2_L_i of an i-th layer of the second image P2 and a Laplacian pyramid of the second image P2.

The Laplacian pyramid of the second image P2 has N+1 layers and comprises a third intermediate image P2_L_top of the second image P2.

The step 404a comprises setting binary mask values of a mask matrix, where the mask matrix represents a weight ratio of the first image P1 that needs to be fused and a weight ratio of the second image P2 that needs to be fused, and performing down-sampling on the mask matrix for N times to obtain a down-sampled matrix mask_down_i, so as to construct an (N+1)-layer Gaussian pyramid of the mask matrix, and performing image mask fusing on P1_L_i and P2_L_i to obtain P12_fusei according to the (N+1)-layer Gaussian pyramid of the mask matrix.

P12_fsei represents a mask fusion image of an i-th layer of the (N+1)-layer Gaussian pyramid of the mask matrix, and an expression thereof is:

$P12_{{\mathrm{fuse}}_i}={\mathrm{mask\_down}}_i*{\mathrm{P1\_L}}_i+{\mathrm{mask\_down}}_i*{\mathrm{P2\_L}}_i.$

* is an image layer fusion operator.

The step 405a comprises reconstructing P12_fusei by following formula to obtain a reconstructed fusion effect image P12_constructi+1 of the i-th layer of the (N+1)-layer Gaussian pyramid of the mask matrix:

$P12_{construc{t}_{i+1}}=\{\begin{array}{cc}\mathrm{UP}\left(P{12}_{{\mathrm{fuse}}_i}\right)*P12_{{\mathrm{fuse}}_{i+1}}& i=0\\ \mathrm{UP}\left(P{12}_{construc{t}_i}\right)*P12_{{\mathrm{fuse}}_{i+1}}& i>0\end{array}.$

UP represents the up-sampling operation, P12_constructi+1 represents the reconstructed fusion effect image of the i-th layer of the (N+1)-layer Gaussian pyramid of the mask matrix, P12_constructi represents a reconstructed fusion effect image of the (i−1)th layer of the (N+1)-layer Gaussian pyramid of the mask matrix, and * is an image layer fusion operator.

The step 406a comprises starting from i=0 to repeatedly execute the step 405a until i=N, constructing a reconstructed fusion effect image P12_constructN+1 of an N-th layer of the (N+1)-layer Gaussian pyramid of the mask matrix, and taking P12_constructN+1 as a final result of image fusion.

In the step 4, in the fusion process, adjusting the quantity of the layers of the Laplace fusion algorithm according to the image quality evaluation function; comprises steps 401b-402b.

The step 401b comprises performing an image splicing quality evaluation on the areas to be spliced and fused, and using a structural similarityinde (SSIM) evaluation method to evaluate an image splicing quality from three aspects of brightness, contrast, and structure similarity.

Expressions of an SSIM evaluation function are:

$S\left(I_{ori},I_{fuse}\right)=\left(l^{\alpha }\left(I_{ori},I_{\mathrm{fuse}}\right),c^{\beta }\left(I_{ori},I_{\mathrm{fuse}}\right),s^{\gamma }\left(I_{ori},I_{\mathrm{fuse}}\right)\right);l\left(I_{ori},I_{\mathrm{fuse}}\right)=\frac{2{\mu }_0{\mu }_f+c_1}{{\mu }_0^2+{\mu }_f^2+c_1};c\left(I_{ori},I_{\mathrm{fuse}}\right)=\frac{2{\sigma }_o{\sigma }_f+c_2}{{\sigma }_o^2+{\sigma }_f^2+c_2};\mathrm{and}$ $s\left(I_{ori},I_{\mathrm{fuse}}\right)=\frac{{\sigma }_{o-f}+c_3}{{\sigma }_o{\sigma }_f+c_3}.$

S(I_ori, I_fuse) is an evaluation coefficient, S(I_ori, I_fuse)∈(0, 1), l^a(I_ori,I_fuse) represents brightness similarity, CR(I_ori,I_fuse) represents contrast similarity, s^y(I_ori, I_fuse) represents structural similarity, I_ori represents an original image, I_fuse represents an image to be evaluated, α, β, γ respectively represent a weighted contribution value on a score of the brightness similarity, a weighted contribution value on a score of the contrast similarity, and a weighted contribution value on a score of the structural similarity. μ_o represents an average intensity of the original image, μ_f represents an average intensity of the image to be evaluated, σ_o represents a pixel standard deviation of the original image, of represents a pixel standard deviation of the image to be evaluated, σ_o-f represents a mutual correlation coefficient between the original image and the image to be evaluated, and C₁, C₂, and C₃ are initial constants.

The step 402b comprises adjusting the quantity of the layers of the Laplace fusion algorithm according to an adjustment function A(N, S) based on the evaluation coefficient.

An expression of the adjustment function is:

$A\left(N,S\right)=20e{\left(S-1\right)}^4+10{\left(S-1\right)}^2+1.$

e is a constant, S is an adjustable parameter, and S∈(0, 1).

The areas A-B, B-A, B-C, C-B, C-D, D-C, D-A, and A-D that need to be spliced and fused are respectively input as the first image P1 and the second image P2 of the Laplace fusion algorithm. Then above operations are performed to obtain four smooth fused images of an upper left smooth fused image, an upper right smooth fused image, a lower right smooth fused image, and a lower left smooth fused image, and the four smooth fused images are spliced with other areas that do not need to be fused to obtain the panoramic surround view that is complete, smooth, and clear.

The step 5 comprises inputting the panoramic surround view into a trained parking space detection model based on deep learning to obtain a parking space attribute result and a parking space identification point result.

The trained parking space detection model uses a repeated visual geometry group network (REP-VGG) network as a network trunk. After the network trunk is output, the network trunk passes through a Sim-CSPSPPF module and residual convolution modules, branches of the network trunk pass through the residual convolution modules, and different parking space results are output on feature maps with different resolutions. The parking space attribute result is predicted by a first feature map with a lower resolution, and the parking space identification point result is predicted by a second feature map with a higher resolution.

The step 5 comprises steps 501-503.

The step 501 comprises providing a total loss function for the trained parking space detection model based on deep learning.

An expression of the total loss function Loss is:

$\mathrm{Loss}={\lambda }_{\mathrm{PKL}}·L_{\mathrm{PKL}}+{\lambda }_{\mathrm{MP}}·L_{\mathrm{MP}}.$

λ_PKL is a weight hyperparameter of a parking space loss function L_PKL, λ_MP is a weight hyperparameter of a parking space identification point loss function L_MP, the parking space loss function L_PKL is designed on the first feature map with the lower resolution, and the parking space identification point loss function L_MP is designed on the second feature map with the higher resolution.

The step 502 comprises setting an initial learning rate to 0.001, selecting AdamW as a gradient descent optimizer, setting BATCH to be 32, setting a quantity of iterations to be 200, and obtaining the trained parking space detection model based on deep learning when a value of the total loss function Loss converges.

The step 503 comprises inputting the panoramic surround view into the trained parking space detection model based on deep learning to obtain information of two parking space identification points of a parking space entrance line, combining the parking space attribute result and the parking space identification point result to obtain coordinate information M1 and M2 of the two parking space identification points of the parking space entrance line, and obtaining coordinate information M3 and M4 of another two parking space identification points of a target parking space that do not appear in the panoramic surround view by parking space identification point derivation expressions.

The parking space identification point derivation expressions are:

$M3=\left[\begin{array}{cc}\cos {\alpha }_n& \sin {\alpha }_n\\ -\sin {\alpha }_n& \cos {\alpha }_n\end{array}\right]\frac{\overrightarrow{M1M2}}{\overrightarrow{M1M2}}{L}_n+M2,$ $\mathrm{and}$ $M4=\left[\begin{array}{cc}\cos {\alpha }_n& \sin {\alpha }_n\\ -\sin {\alpha }_n& \cos {\alpha }_n\end{array}\right]\frac{\overrightarrow{M1M2}}{\overrightarrow{M1M2}}{L}_n+M1.$

α_n represents an angle of an n-th parking space in the panoramic surround view, and L_n represents a depth of the n-th parking space in the panoramic surround view.

An expression of the parking space loss function L_PKL is:

$L_{\mathrm{PKL}}=W_T·L_T+W_O·L_O+W_C·L_C.$

W_T represents a weight hyperparameter of a parking space category loss function L_T, W_o represents a weight hyperparameter corresponding to a parking space occupancy loss function L_o, and W_c represents a weight hyperparameter corresponding to a parking location confidence loss function L_C.

Expressions of the parking space category loss function L_T are:

$L_T={\sum }_{g=1}^{w1·h1}{P}_T^i·-\frac{1}{3}{\sum }_{t=1}^3\left[y_t·\log \left(\sigma \left(x_t\right)\right)+\left(1-y_t\right)·\log \left(1-\sigma \left(x_t\right)\right)\right],$ $\mathrm{and}$ $\sigma \left(x_t\right)=\frac{1}{1+\exp \left(-x_t\right)}.$

w1 represents a width of the first feature map, h1 represents a height of the first feature map, PT determines whether a center of a g-th grid belongs to a parking space area, P_Tⁱ=1 represents that the center of the g-th grid belongs to the parking space area, and P_Tⁱ=0 represents that the center of the g-th grid does not belong to the parking space area. x_t represents a probability that a predicted parking space is a t-th type parking space. y_t represents a true value that the predicted parking space belongs to the t-th type parking space, y_t=(0, 0, 1) represents that the predicted parking space is a vertical parking space, y_t=(0, 1, 0) represents that the predicted parking space is a parallel parking space, y_t=(1, 0, 0) represents that the predicted parking space is an inclined parking space, and σ(x_t) is an intermediate function.

An expression of the parking space occupancy loss function L_O is:

$L_O={\sum }_{g=1}^{w1·h1}\left[\alpha ·{P_A^g\left(S_{o_x}^g-S_{o_y}^g\right)}^2+{P_{\mathrm{IN}}^g\left(S_{o_x}^g-S_{o_y}^g\right)}^2\right].$

α is a weight value for balancing valid parking spaces and invalid parking spaces, P_A^g determines whether the center of the g-th grid is within a valid parking space area, P_A^g=1 represents that the center of the g-th grid is within the valid parking space area, and P_A^g=0 represents that the center of the g-th grid is located in a non-valid parking space area. P_IN^g determines whether the g-th grid is located in an invalid parking space area, P_IN^g=1=1 represents that the center of the g-th grid is located in the invalid parking space area, and P_IN^g=0 indicates that the center of the g-th grid is located in a non-invalid parking space area. S_ox^g represents a probability of whether the predicted parking space on the g-th grid is valid, S_oy^g represents a true value of whether the predicted parking space on the g-th grid is valid, S_oy^g=0 represents that the predicted parking space is a valid parking space, and S_oy^g=1 represents that the predicted parking space is an invalid parking space.

An expression of the parking location confidence loss function L_C is:

$L_C={\sum }_{g=1}^{w1·h1}\left[P_C^g·{\left(C_x^g-C_y^g\right)}^2+{\alpha }_P\left(1-P_C^g\right){\left(C_x^g-C_y^g\right)}^2\right].$

P_C^g determines whether the center of the g-th grid belongs to the parking space area, P_C^g=1 represents that the center of the g-th grid belongs to the parking space area, and P_C^g=0 represents that the center of the g-th grid does not belong to the parking space area. C_x^g represents a predicted probability value that the g-th grid belongs to the parking space. C_y^g represents a true value that the g-th grid belongs to the parking space, and α_p is a weight value for balancing the parking space area and a non-parking space area.

An expression of the parking space identification point loss function L_MP is:

$L_{\mathrm{MP}}=W_{M_{\mathrm{xy}}}·L_{M_{\mathrm{xy}}}+W_{\theta }·L_{\theta }.$

W_Mxy represents a weight hyperparameter corresponding to a parking space identification point coordinate loss function L_Mxy, and W_θ represents a weight hyperparameter corresponding to a parking space identification point angle orientation loss function L_θ.

An expression of the parking space identification point coordinate loss function L_Mxy is:

$L_{M_{\mathrm{xy}}}={\sum }_{g=1}^{w2·h2}{P}_{M_{\mathrm{xy}}}^g·\left[{\left(X_x^g-\frac{X_y^g}{w2}\right)}^2+{\left(Y_x^g-\frac{Y_y^g}{h2}\right)}^2\right].$

w2 represents a width of the second feature map, h2 represents a height of the second feature map, P_Mxy determines whether a g-th grid contains one of the parking space identification points, P_Mxy^g=1 represents that the g-th grid contains the one of the parking space identification points, and P_Mxy^g=0 represents that the g-th grid does not contain the one of the parking space identification points. X_x^g represents a predicted value of the one of the parking space identification points in an x-coordinate direction of the g-th grid, X_y^g represents a true value of the one of the parking space identification points in the x-coordinate direction of the g-th grid. Y_x^g represents a predicted value of the one of the parking space identification points in a y-coordinate direction of the g-th grid, and Y_y^g represents a true value of the one of the parking space identification points in the y-coordinate direction of the g-th grid.

An expression of the parking space identification point angle orientation loss function L_θ is:

$L_{\theta }={\sum }_{g=1}^{w2·h2}{P}_{M_{\mathrm{xy}}}^g·\left[{\left(\cos {\theta }_x^g-\cos {\theta }_y^g\right)}^2+{\left(\sin {\theta }_x^g-\sin {\theta }_y^g\right)}^2\right].$

cosθ_x^g represents a predicted cosine value of an angle orientation of the one of the parking space identification points on the g-th grid, cosθ_y^g represents a true cosine value of the angle orientation of the one of the parking space identification points on the g-th grid. sinθ_x^g represents a predicted sine value of the angle orientation of the one of the parking space identification points on the g-th grid, and sinθ_y^g represents a true sine value of the angle orientation of the one of the parking space identification points on the g-th grid.

Finally, by combining information of the four parking space identification points of each of the parking spaces in the panoramic surround view and the parking space attribute information, detailed information of all of the parking spaces around the automobile is obtained in real time, and the detailed information of all of the parking spaces is transmitted to a next level of autonomous driving path planning module and a decision-making control module to realize an automatic parking function of the automobile.

In summary, the unmanned parking space detection method based on the panoramic surround view of the present disclosure has following characteristics.

The unmanned parking space detection method provided by the present disclosure only requires four cameras to complete tasks of image acquisition and splicing, and cost thereof is inexpensive.

The present disclosure adopts the Laplace fusion algorithm to splice the front view, the rear view, the left view, and the right view to obtain the panoramic surround view, which effectively realize fusion of the front view, the rear view, the left view, and the right view. By using the image quality evaluation function, complexity of the unmanned parking space detection method is reduced while ensuring a quality of the splicing. The panoramic surround view generated by the present disclosure has a more natural transition in the areas being spliced and fused, and the panoramic surround view display is more complete and beautiful.

Compared with conventional machine vision detection technologies, the unmanned parking space detection method based on the panoramic surround view of the present disclosure adopts a target detection technology for detecting parking space entrance lines, which has higher detection accuracy and stronger robustness. The present disclosure adopts a more lightweight network design to ensure real-time performance of a detection algorithm thereof.

The unmanned parking space detection method based on the panoramic surround view of the present disclosure does not rely on relative spatial locations of vehicles around a target parking space, but only requires a presence of the parking entrance lines on the ground in a parking area that meets parking standards to accomplish a parking space detection task.

The unmanned parking space detection method based on the panoramic surround view of the present disclosure is applicable to a wide range of parking environments, and is able to detect whether the target parking space is a vertical parking space, a parallel parking space, or an inclined parking space, and is able to obtain information about whether the target parking space is occupied.

In the description of the present disclosure, the description of reference terms “one embodiment”, “some embodiments”, “examples”, “particular examples”, “some examples”, etc. mean that particular features, structures, materials, or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present disclosure. In the specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present disclosure have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principle and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims and their equivalents.

Claims

1. An unmanned parking space detection method based on a panoramic surround view, comprising: a step 1: respectively acquiring a front original image, a rear original image, a left original image, and a right original image of a surrounding environment of an automobile through a front-view camera, a rear-view camera, a left-view camera and a right-view camera arranged on the automobile;a step 2: respectively performing distortion removal processing and perspective transformation processing on the front original image, the rear original image, the left original image, and the right original image to obtain a front view, a rear view, a left view, and a right view;a step 3: extracting feature points from the front view, the rear view, the left view and the right view by a random sample consensus (RANSAC) algorithm, and determining areas to be spliced and fused;a step 4: fusing the areas to be spliced and fused by a Laplace fusion algorithm to obtain the panoramic surround view, and adjusting a quantity of layers of a Laplace pyramid according to an image quality evaluation function in a fusion process; anda step 5: inputting the panoramic surround view into a trained parking space detection model based on deep learning to obtain a parking space attribute result and a parking space identification point result;wherein the trained parking space detection model based on deep learning uses a repeated visual geometry group network (REP-VGG) network as a network trunk, and after the network trunk is output, the network trunk passes through a Sim-CSPSPPF module and residual convolution modules, branches of the network trunk pass through the residual convolution modules, and different parking space results are output on feature maps with different resolutions; the parking space attribute result is predicted by a first feature map with a lower resolution, and the parking space identification point result is predicted by a second feature map with a higher resolution.

2. The unmanned parking space detection method based on the panoramic surround view according to claim 1, wherein the step 2 comprises: a step 201: converting physical coordinates in a world coordinate system to pixel coordinates in a pixel coordinate system by following formula:

3. The unmanned parking space detection method based on the panoramic surround view according to claim 1, wherein in the step 4, fusing the areas to be spliced and fused by the Laplace fusion algorithm to obtain the panoramic surround view comprises: a step 401a: performing Gaussian smoothing and down-sampling on a first image P1 in two images to be spliced and fused and performing Gaussian smoothing and up-sampling on the first image P1, and respectively repeating for N times to obtain an (N+1)-layer Gaussian pyramid of the first image P1, obtaining a first intermediate image P_downiof an i-th layer of the (N+1)-layer Gaussian pyramid during an i-th Gaussian smoothing and down-sampling, obtaining a second intermediate image P_upi of the i-th layer of the (N+1)-layer Gaussian pyramid during an i-th Gaussian smoothing and up-sampling, i≤N, and the (N+1)-layer Gaussian pyramid of the first image P1 comprises N second intermediate images P_upiand the first image P1;a step 402a: constructing the Laplace pyramid of the first image P1 by following formula:

4. The unmanned parking space detection method based on the panoramic surround view according to claim 3, wherein in the step 4, in the fusion process, adjusting the quantity of the layers of the Laplace fusion algorithm according to the image quality evaluation function; comprises: a step 401b: performing an image splicing quality evaluation on the areas to be spliced and fused, and using a structural similarityinde (SSIM) evaluation method to evaluate an image splicing quality from three aspects of brightness, contrast, and structure similarity, and expressions of an SSIM evaluation function are:

5. The unmanned parking space detection method based on the panoramic surround view according to claim 1, wherein the step 5 comprises: a step 501: providing a total loss function for the trained parking space detection model based on deep learning, where an expression of the total loss function Loss is:

6. The unmanned parking space detection method based on the panoramic surround view according to claim 5, wherein an expression of the parking space loss function LPKL is:

7. The unmanned parking space detection method based on the panoramic surround view according to claim 5, wherein an expression of the parking space identification point loss function LMP is: