Patent Application
20250002007
The present disclosure relates to an information processing device, an information processing method, and a program.
Specifically, the present disclosure relates to an information processing device, an information processing method, and a program that enable, for example, processing of identifying parking availability of each of a plurality of parking spots in a parking lot, an entrance direction of each parking spot, and the like, and generating display data based on an identification result to display the display data on a display unit, and performing automated parking based on the identification result.
For example, in many parking lots in shopping centers, amusement parks, sightseeing spots, other places in a town, or the like, a large number of vehicles can park in many cases.
A user who is a driver of the vehicle searches the parking lot for a vacant space where the driver can park the vehicle and parks the vehicle. In this case, the user travels the vehicle in the parking lot, visually checks around, and searches for a vacant space.
Such processing for checking a parkable space needs time, and furthermore, there is a problem in that, if the vehicle travels in the narrow parking lot, a contact accident with another vehicle or person is likely to occur.
As one technique for solving this problem, for example, there is a technique of analyzing an image captured by a camera provided in a vehicle (automobile), detecting a parkable parking spot, and displaying detection information on a display unit in the vehicle.
In a configuration in which parking spot analysis processing of a vacant parking spot or the like based on an image captured by the camera is performed, there is a configuration in which a top surface image (overhead view image) viewed from above the vehicle is generated and used.
The top surface image (overhead view image) can be generated by, for example, composite processing using images captured by multiple cameras that individually capture front, rear, left, and right directions of the vehicle.
However, in such a composite image, it may be difficult to determine a subject object due to distortion or the like generated at the time of image composition. As a result, even if the top surface image as the composite image is analyzed, it may not be possible to accurately identify an occupied parking spot in which a parked vehicle is present and a vacant parking spot in which no parked vehicle is present.
Furthermore, recently, technology development related to automated driving and driving support has been actively conducted. For example, advanced driver assistance system (ADAS), autonomous driving (AD) technology, and the like are used.
However, even in a case where automated parking using automated driving or driving support is performed, it is necessary to perform processing of detecting a parkable parking spot from a parking lot and processing of detecting an entrance for each parking spot, and the processing of these are performed using, for example, an image captured by a camera provided in a vehicle (automobile).
Therefore, for example, in a case where an image in which it is difficult to determine a subject object as described above is used, there is a problem that it is difficult to accurately identify the occupied parking spot and the vacant parking spot, and smooth automated parking becomes impossible.
Note that, as a related art that discloses a configuration for detecting a parkable region on the basis of an image captured by a camera, there is, for example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2020-123343).
This Patent Document 1 discloses technology of detecting two diagonal feature points of a parking spot from an image captured by a camera provided in a vehicle, estimating a center position of the parking spot by using a line segment connecting the two detected diagonal feature points, and estimating a region of the parking spot on the basis of the estimated center point position of the parking spot.
However, this related art is based on the premise that two diagonal feature points of one parking spot are detected from an image captured by the camera, and there is a problem that analysis cannot be performed in a case where the two diagonal feature points of the parking spot cannot be detected.
The present disclosure is to solve the above-described problems, for example, and an object thereof is to provide an information processing device, an information processing method, and a program capable of estimating a range or a state (vacant/occupied) of each parking spot even in a state where it is difficult to directly identify a region or a state (vacant/occupied) of a parking spot from an image captured by a camera.
Specifically, an object is to provide an information processing device, an information processing method, and a program capable of processing of identifying parking availability of each of a plurality of parking slots in a parking lot, an entrance direction of each parking spot, and the like by using a learning model generated in advance, and generating display data based on an identification result to display the display data on a display unit, and performing automated parking based on the identification result.
A first aspect of the present disclosure is an information processing device including:
Moreover, a second aspect of the present disclosure is an information processing method executed in an information processing device, in which the information processing device includes a parking spot analysis unit configured to execute analysis processing on a parking spot included in an image, and the parking spot analysis unit estimates a parking spot definition rectangle indicating a parking spot region in the image, by using a learning model generated in advance.
Moreover, a third aspect of the present disclosure is a program for executing information processing in an information processing device, in which the information processing device includes a parking spot analysis unit configured to execute analysis processing on a parking spot included in an image, and the program causes the parking spot analysis unit to estimate a parking spot definition rectangle indicating a parking spot region in the image, by using a learning model generated in advance.
Note that the program of the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium that provides the program in a computer-readable format, to an information processing device, an image processing device, or a computer system capable of executing the program codes. By providing such a program in a computer-readable format, processing corresponding to the program is implemented on the information processing device or the computer system.
Other purposes, features, and advantages of the present disclosure would be obvious by the detailed description based on the embodiments of the present invention as described later and the attached drawings. Note that a system described herein is a logical set configuration of a plurality of devices, and is not limited to a system in which devices with respective configurations are in the same housing.
According to a configuration of an embodiment of the present disclosure, a configuration is realized in which a learning model is applied to estimate a parking spot definition rectangle (polygon), a parking spot entrance direction, and a vacancy state of a parking spot.
Specifically, for example, a top surface image generated by combining individual images captured by front, rear, left, and right cameras mounted on a vehicle is analyzed, and analysis processing is performed on a parking spot in the image. The parking spot analysis unit uses a learning model to estimate a vertex of a parking spot definition rectangle (polygon) indicating a parking spot region in the image and an entrance direction of the parking spot. Furthermore, estimation is performed as to whether the parking spot is a vacant parking spot or an occupied parking spot in which a parked vehicle is present. The parking spot analysis unit uses CenterNet as a learning model to execute, for example, estimation processing on a spot center and a vertex of a parking spot definition rectangle (polygon).
With this configuration, a configuration is realized in which a learning model is applied to estimate a parking spot definition rectangle (polygon), a parking spot entrance direction, and a vacancy state of a parking spot.
Note that the effects described herein are merely examples and are not limited, and additional effects may also be provided.
Hereinafter, an information processing device, an information processing method, and a program according to the present disclosure will be described in detail with reference to the drawings. Note that the description will be made according to the following items.
First, an outline of processing executed by an information processing device according to the present disclosure will be described.
The information processing device according to the present disclosure is, for example, a device mounted on a vehicle, and analyzes an image captured by a camera provided in the vehicle or a composite image thereof by using a learning model generated in advance, to detect a parking spot in a parking lot. Furthermore, it is identified whether the detected parking spot is a vacant parking spot which is vacant or an occupied parking spot in which there is already a parked vehicle, and an entrance direction of each parking spot is identified.
Moreover, in an embodiment of the information processing device according to the present disclosure, processing of generating display data based on identification results of these and displaying the display data on a display unit is performed, and automated parking processing and the like based on the identification results are performed.
With reference to
The vehicle 10 may be a general manual driving vehicle driven by a driver, an automated driving vehicle, or a vehicle having a driving support function.
The automated driving vehicle or the vehicle having the driving support function is, for example, a vehicle equipped with an advanced driver assistance system (ADAS) or an autonomous driving (AD) technology. These vehicles can perform automated parking using automated driving and driving support.
The vehicle 10 illustrated in
With reference to
As illustrated in
It is possible to generate an image observed from above the vehicle 10, that is, a top surface image (overhead view image) by combining four images individually captured by the cameras that capture images in four directions of front, rear, left, and right of the vehicle 10.
The display data displayed on the display unit 12 illustrated in
The example of the display data illustrated in
As described above with reference to
As a result, an object displayed on the top surface image displayed on the display unit 12 of the vehicle 10 may be displayed as an image having distortion and a shape different from a shape of an actual object. Specifically, vehicles, parking spot lines, and the like in the parking lot are displayed in shapes different from actual shapes.
Data displayed on the display unit 12 illustrated in
Among parking spots around the own vehicle, a white line indicating a parking spot is clearly displayed in a part of a parking spot on the left side of the own vehicle, but an object estimated as a parked vehicle is deformed and displayed in a parking spot on the right side of the own vehicle and in a parking lot on the left rear side of the own vehicle.
For example, in a case where such an image is displayed on the display unit 12 of the vehicle 10, it is difficult for the driver to accurately identify whether or not an object displayed in the parking spot is a parked vehicle, and it is also difficult for the driver to clearly identify a boundary of each parking spot, and a vacant state, an occupied state, and the like of each parking spot.
As a result, in many cases, the driver gives up checking from the display image, and performs processing of checking the front of the vehicle while driving and searching for an available parking spot again.
Furthermore, in a case where the vehicle is an automated driving vehicle and is a vehicle capable of performing automated parking processing, an image with many deformations as illustrated in
However, it is also difficult for the automated driving control unit to identify whether or not a displayed object in the parking spot is a parked vehicle from the input image, and it is also difficult to clearly identify a boundary of each parking spot, and a vacancy state, an occupied state, and the like of each parking spot. As a result, automated parking cannot be performed in some cases.
Note that, in the image illustrated in
In some the parking lots, a parking direction may be defined. However, it is impossible to determine a front-rear direction of a parked vehicle from the image as illustrated in
Note that an example of a composite image that does not include the entire parking spot is, for example, an image as illustrated in
The information processing device according to the present disclosure, that is, the information processing device mounted on a vehicle is to solve such a problem, for example.
The information processing device according to the present disclosure performs parking spot detection in a parking lot by performing image analysis using a learning model generated in advance. Moreover, it is identified whether a detected parking spot is a vacant parking spot that is vacant or an occupied parking spot that already has a parked vehicle, and an entrance direction of each parking spot is identified.
Moreover, processing of generating display data based on identification results of these and displaying the display data on the display unit is performed, and automated parking processing and the like based on the identification results are performed.
With reference to
The display data of the display unit 12 illustrated in
The display data illustrated in
The information processing device according to the present disclosure superimposes and displays a parking spot identification frame on the top surface image.
The parking spot identification frame to be superimposed and displayed has a rectangular (polygon) shape including four vertexes that define a region of each parking spot.
Moreover, a vacant parking spot identification frame indicating a vacant parking spot in which no parked vehicle is present and an occupied parking spot identification frame indicating an occupied parking spot in which a parked vehicle is present are displayed in different display modes.
Specifically, for example, the vacant parking spot identification frame and the occupied parking spot identification frame are displayed as frames of different colors such as a “blue frame” and a “red frame”, respectively.
Note that the color setting is an example, and various other color combinations are possible.
Moreover, the information processing device according to the present disclosure superimposes and displays a parking spot entrance direction identifier indicating an entrance direction (vehicle entry direction) of each parking spot on the top surface image of the parking lot.
The example illustrated in the figure is an example in which an “arrow” is used as the parking spot entrance direction identifier.
Note that, as the parking spot entrance direction identifier, various different identifiers can be used in addition to the “arrow”.
For example, one edge on an entrance side of the parking spot identification frame is displayed as a different color (for example, white). Alternatively, various display modes are possible, such as displaying two vertexes on the entrance side of the parking spot identification frame with a different color (for example, white).
The display data illustrated in
Moreover, as illustrated in
In this manner, the identification tag (state (vacant/occupied) identification tag) indicating whether the parking lot is vacant or occupied may be displayed in each parking spot.
As described with reference to
For example, in a case of a manual driving vehicle, the driver can reliably and easily determine a vacant or occupied state and an entrance direction of each parking frame, on the basis of the above-described identification data displayed on the display unit.
Furthermore, in a case of an automated driving vehicle, an image (top surface image) to which the identification data described above is added is input to the automated driving control unit. The automated driving control unit can reliably and easily determine a vacant or occupied state and an entrance direction of each parking frame on the basis of these pieces of identification data, and can perform automated parking processing accompanied by highly accurate position control for the vacant parking spot.
Note that examples of the display data illustrated in
As described above with reference to
As illustrated in
By superimposing and displaying these pieces of identification data, it is possible to easily and reliably identify a region of each parking spot, a state (vacant/occupied) of each parking spot, and an entrance direction of each parking spot.
By generating such identification data and displaying the identification data on the display unit or supplying the identification data to the automated driving control unit, safe and reliable parking processing can be performed in both the manual driving vehicle and the automated driving vehicle.
Next, a description will be given to an outline of learning model generation processing and parking spot analysis processing applied with a learning model, executed by the information processing device of the present disclosure.
As described above, the information processing device of the present disclosure is a device mounted on a vehicle, and analyzes an image captured by a camera provided in the vehicle or a composite image thereof by using a learning model generated in advance, to execute analysis processing on a parking spot in a parking lot.
Specifically, by using the learning model generated in advance, whether the parking spot is a vacant parking spot or an occupied parking spot in which a parked vehicle is present is identified, and an entrance direction of each parking spot is further identified.
Moreover, processing of generating display data based on identification results of these and displaying the display data on the display unit is performed, and automated parking processing and the like based on the identification results are performed.
With reference to
As illustrated in
The top surface image (composite image) is a composite image generated using images captured by multiple cameras that capture the front, rear, left, and right of the vehicle 10, and corresponds to an image observed from above the vehicle 10.
Note that the top surface image (composite image) illustrated on the left side of
Furthermore, the identification data to be superimposed on the output image on the right side of
Note that the output image on which the identification data is superimposed illustrated on the right side of
As illustrated in
With reference to
“(1) The input image (top surface image (composite image))” is an image similar to the input image on the left side of
The parking spot analysis unit 120 of the information processing device 100 of the present disclosure analyzes this input image, and generates the following identification data illustrated on the right side of
“(a) The vacant parking spot correspondence identification data” in
Furthermore, “(b) the occupied parking spot correspondence identification data” of
Note that “parking spot definition polygon four vertexes” illustrated in each of
By connecting the four vertexes of the polygon, it is possible to draw the vacant parking spot identification frame and the occupied parking spot identification frame.
That is, the parking spot analysis unit 120 of the information processing device 100 of the present disclosure calculates positions (coordinates) of four vertexes constituting the rectangle (polygon) that defines a region of each parking spot, and draws the vacant parking spot identification frame and the occupied parking spot identification frame.
The parking spot analysis unit 120 of the information processing device 100 of the present disclosure is input with a top surface image (composite image) as illustrated on the left side of
In order to generate these pieces of identification data, the learning model 180 generated in advance is used.
With reference to
The learning processing unit 80 is input with a large number of pieces of learning data (teacher data) as illustrated on the left side of
As the learning data (teacher data), for example, specifically, teacher data including set data in which various top surface images (composite images) of a parking lot and parking spot information corresponding to each of parking spots in the images are added as annotations (metadata) is used.
That is, the learning processing unit 80 is input with a large number of top surface images (composite images) of the parking lot to which parking spot information analyzed in advance has been added as annotations, and executes the learning processing using these as teacher data.
The learning model 180 generated by the learning processing is, for example, the learning model 180 that is input with a top surface image of a parking lot and outputs parking spot information as an output.
Note that the number of learning models is not limited to one, and multiple learning models in units of processing can be generated and used. For example, a learning model corresponding to processing as described below can be generated and used.
The parking spot analysis unit 120 of the information processing device 100 of the present disclosure generates, for example, the following parking spot information by using the generated learning model 180.
The learning processing unit 80 illustrated in
The learning data (teacher data) input to the learning processing unit 80 includes an image and an annotation (metadata) that is additional data corresponding to the image. The annotation is parking spot information analyzed in advance.
With reference to
As illustrated in
The learning data (teacher data) includes these annotations, that is, metadata analyzed in advance, and is input to the learning processing unit 80 together with the image.
Note that the top surface image (composite image) input to the learning processing unit 80 also includes an image in which the entire parking spot is not captured in the image. For example, in the parking lot image illustrated in the learning data illustrated on the left side of
Also for such an image, the region of each parking spot is examined in advance, each of the vertex coordinates of polygon defining the parking spot is obtained, teacher data associated with each image is generated as an annotation, and the learning processing is executed.
For example, in the top surface image illustrated in
By executing the parking spot analysis using the learning model 180 generated by performing such learning processing, it becomes possible to perform processing of estimating the rectangle (polygon) defining the region of the parking spot even in a case where only a part of the parking spot is captured in the top surface image of the parking lot to be an analysis target.
Next, a description will be given to a configuration of the parking spot analysis unit of the information processing device of the present disclosure and details of the parking spot analysis processing executed by the parking spot analysis unit.
The parking spot analysis unit 120 of the information processing device 100 of the present disclosure is input with, for example, a top surface image generated by combining images captured by four cameras that capture images in four directions of front, rear, left, and right of the vehicle, analyzes a parking spot included in the input top surface image, and generates parking spot information corresponding to each parking spot as an analysis result.
The generated parking spot information corresponding to each parking spot includes, for example, the following identification data, namely:
As illustrated in
The parking spot configuration estimation unit 123 includes a spot center grid estimation unit 131, a spot center relative position estimation unit 132, a spot vertex relative position and entrance estimation first algorithm execution unit 133, a spot vertex relative position and entrance estimation second algorithm execution unit 134, and a spot vertex pattern estimation unit 135.
Furthermore, the estimation result analysis unit 124 includes a parking spot state (vacant/occupied) determination unit 141, a spot vertex relative position and entrance estimation result selection unit 142, a rescale unit 143, a parking spot center coordinate calculation unit 144, a parking spot definition polygon vertex coordinate calculation unit 145, and a parking spot definition polygon coordinate rearrangement unit 146.
Hereinafter, processing executed by each component of the parking spot analysis unit 120 will be sequentially described.
The feature amount extraction unit 121 extracts a feature amount from a top surface image that is an input image.
The feature amount extraction unit 121 executes feature amount extraction processing, by using one learning model generated by the learning processing unit 80 described above with reference to
That is, the feature amount extraction processing using a learning model for performing the feature amount extraction processing from an image is executed.
Specifically, for example, feature amount extraction using Resnet-18, which is a learning model configured by an 18-layer convolutional neural network (CNN), is executed.
By using the Resnet-18 (CNN) generated by the learning processing using a large number of parking lot images including a vacant parking spot having no parked vehicle and an occupied parking spot having a parked vehicle, it becomes possible to extract various feature amounts that can be used to identify whether each of the parking spots included in the input image is a vacant parking spot or an occupied parking spot.
Note that the feature amount extraction unit 121 is not limited to the Resnet-18 (CNN), and configurations using various other feature amount extraction means and feature amount extraction learning models can be used.
The feature amount extracted from the image by the feature amount extraction unit 121 includes a feature amount that can be used for region determination of the parking spot in the image, state (vacant/occupied) determination of each parking spot, and entrance direction determination processing of each parking spot.
The parking lot image contains, as subjects, various objects such as a white line defining a parking spot or a vehicle stop block, a wall and a pillar of the parking lot, and a vehicle parked in the parking spot, and feature amounts corresponding to these various objects are extracted.
The feature amount data extracted from the image by the feature amount extraction unit 121 is input to the parking spot configuration estimation unit 123 together with image data via the down-sampling unit 122.
The down-sampling unit 122 executes down-sampling processing on the input image (top surface image) and the feature amount data extracted from the input image (top surface image) by the feature amount extraction unit 121. Note that the down-sampling processing is performed to reduce a processing load in the parking spot configuration estimation unit 123, and is not essential.
The parking spot configuration estimation unit 123 is input with the input image (top surface image) and the feature amount data extracted from the image by the feature amount extraction unit 121, and executes analysis processing on a configuration, a state (vacant/occupied), and the like of the parking spot included in the input image.
The learning model 180 generated by the learning processing unit 80 described above with reference to
The learning model used by the parking spot configuration estimation unit 123 is, for example, learning models as follows:
As described above, the parking spot configuration estimation unit 123 includes the spot center grid estimation unit 131, the spot center relative position estimation unit 132, the spot vertex relative position and entrance estimation first algorithm execution unit 133, the spot vertex relative position and entrance estimation second algorithm execution unit 134, and the spot vertex pattern estimation unit 135.
Hereinafter, details of processing executed by each of these components will be sequentially described.
Processing executed by the spot center grid estimation unit 131 will be described with reference to
“(1) The grid setting example for an input image” illustrated in
The feature amount extracted by the feature amount extraction unit 121 described above can be analyzed as a feature amount on a grid basis in the spot center grid estimation unit 131, and the spot center grid estimation unit 131 can perform estimation processing on a center grid of each parking spot on the basis of the feature amount on a grid basis.
As illustrated on the right side of
Data (a1) and (b1) illustrated on the left side of
Data (a2) and (b2) illustrated on the right side of
As described above, spot center grid estimation processing in the spot center grid estimation unit 131 uses the learning model 180 generated by the learning processing unit 80 described above with reference to
Specifically, for example, processing using a learning model called “CenterNet” is possible.
The “CenterNet” is a learning model that enables estimation of a region of the entire object by analyzing center positions of various objects and calculating an offset from the center position to an end point of the object.
As an object region estimation technique, a technique using “bounding box” has been often used so far.
The “CenterNet” is a technique capable of performing region estimation of an object more efficiently than the “bounding box”.
With reference to
The “bounding box” is a technique of estimating a quadrangle surrounding the object (bicycle) 201.
However, in order to determine the “bounding box” which is the quadrangle surrounding the object, there is a problem that it is necessary to perform processing of selecting the most probable bounding box from a large number of bounding boxes set on the basis of an object existence probability or the like according to a shape and a state of the object, and the processing efficiency is poor.
On the other hand, the object region estimation technique using the “CenterNet” is to estimate a center position of the object, and thereafter, processing is performed in which the quadrangle (polygon) surrounding the object is estimated by estimating a relative position of vertexes of a rectangle (polygon) that defines the object region from the estimated object center.
The object region estimation technique using the “CenterNet” can estimate the quadrangle (polygon) surrounding the object more efficiently than the “bounding box”.
Note that, in the “CenterNet”, an object center identification heat map is generated in order to estimate an object center position.
An example of generation processing for the object center identification heat map will be described with reference to
The object center identification heat map is input to, for example, a convolutional neural network (CNN) for object center detection, which is a learning model in which an object image is generated in advance.
The object center detection convolutional neural network (CNN) is a CNN (learning model) generated by learning processing of a large number of images of objects in the same category, that is, a large number of various bicycles in the example illustrated in the figure.
An image to be an object center analysis target, that is, (1) an object image illustrated in
Note that, in (2) the object center identification heat map illustrated in the figure, a bright portion corresponds to the peak region, and is a region having a high probability of being the object center.
A position of an object center grid can be determined as illustrated in
Note that, in the processing of the present disclosure, the analysis target object is a parking spot, and the object center estimated by the spot center grid estimation unit 131 is a parking spot center.
That is, as illustrated in
With reference to
A spot center grid is estimated for one vacant parking spot in (1) an input image (top surface image) illustrated in the lower left of
A spot center grid is estimated for (a1) a spot center estimation target parking spot (vacant parking spot) illustrated on the left of
The spot center grid estimation unit 131 inputs, to the learning model (CNN), image data of (a1) the spot center estimation target parking spot (vacant parking spot) illustrated on the left in
Note that the learning model (CNN) used here is two learning models (CNN) as illustrated in the figure. Namely, to the following two learning models (CNN), which are:
Here, “(m1) the CNN for vacant class correspondence spot center detection” is a learning model (CNN) generated by learning processing using, as teacher data, a large number of images of various vacant parking spots, that is, a large number of images of vacant parking spots (with annotations for spot centers) in which no vehicle is parked. That is, it is a convolutional neural network (CNN) for vacant parking spot center detection for estimating a spot center in a vacant parking spot.
Whereas, “(m2) the CNN for occupied class correspondence spot center detection” is a learning model (CNN) generated by learning processing using, as teacher data, a large number of images of various occupied parking spots, that is, a large number of images of occupied parking spots (with annotations of spot centers) in which various vehicles are parked. That is, it is a convolutional neural network (CNN) for occupied parking spot center detection for estimating a spot center in an occupied parking spot.
Two heat maps illustrated at the right end of
Namely, two heat maps of:
In the two heat maps illustrated in the figure, a peak (output value) illustrated in the center portion of “(a2) the spot center identification heat map generated by applying the vacant class correspondence learning model (CNN)” on the upper side is larger than a peak (output value) shown at a center portion of “(a3) the spot center identification heat map generated by applying the occupied class correspondence learning model (CNN)” on the lower side.
This is due to similarity between an object (vacant parking spot) as a spot center determination target and an object class of the used learning model (CNN).
That is, in the processing example illustrated in
In this case, “(m1) the CNN for vacant class correspondence spot center detection” which is the learning model (CNN) generated on the basis of images of vacant parking spots has higher object similarity with (a1) the spot center estimation target parking spot (vacant parking spot).
Whereas, “(m2) the CNN for occupied class correspondence spot center detection” which is the learning model (CNN) generated on the basis of images of occupied parking spots has lower object similarity with (a1) the spot center estimation target parking spot (vacant parking spot), so that a heat map with a small peak is generated.
These two spot center identification heat maps having the different peaks are input to the parking spot state (vacant/occupied) determination unit 141 of the estimation result analysis unit 124 of the parking spot analysis unit 120 illustrated in
The parking spot state (vacant/occupied) determination unit 141 determines that the learning model (CNN) on a side that has output the spot center identification heat map having a large peak is close to a state of the parking spot as a parking spot state (vacant/occupied) determination target.
For example, in the example illustrated in
In this case, the parking lot as a parking spot state (vacant/occupied) determination target is determined to be a vacant parking spot in which no parked vehicle is present.
Note that this processing will be described again later.
As described with reference to
Moreover, as illustrated in
As illustrated
The spot center grid estimation processing example described with reference to
Next, with reference to
A spot center grid is estimated for one occupied parking spot in (1) an input image (top surface image) illustrated in the lower left of
A spot center grid is estimated for (b1) a spot center estimation target parking spot (occupied parking spot) illustrated on the left of
The spot center grid estimation unit 131 inputs, to the learning model, image data of (b1) the spot center estimation target parking spot (occupied parking spot) illustrated on the left in
The learning model used here is two learning models (CNN) similarly to that described above with reference to
As described above, “(m1) the CNN for vacant class correspondence spot center detection” is a learning model (CNN) generated by learning processing using, as teacher data, a large number of images of various vacant parking spots, that is, a large number of images of vacant parking spots (with annotations for spot centers) in which no vehicle is parked. That is, it is a convolutional neural network (CNN) for vacant parking spot center detection for estimating a spot center in a vacant parking spot.
Whereas, “(m2) the CNN for occupied class correspondence spot center detection” is a learning model (CNN) generated by learning processing using, as teacher data, a large number of images of various occupied parking spots, that is, a large number of images of occupied parking spots (with annotations of spot centers) in which various vehicles are parked. That is, it is a convolutional neural network (CNN) for occupied parking spot center detection for estimating a spot center in an occupied parking spot.
Two heat maps illustrated at the right end of
Namely, two heat maps of:
In the two heat maps illustrated in the figure, a peak (output value) illustrated in a center portion of “(b2) the spot center identification heat map generated by applying the vacant class correspondence learning model (CNN)” on the upper side is smaller than a peak (output value) shown at a center portion of “(b3) the spot center identification heat map generated by applying the occupied class correspondence learning model (CNN)” on the lower side.
This is due to similarity between an object (occupied parking spot) as a spot center determination target and an object class of the used learning model (CNN).
That is, in the processing example illustrated in
In this case, “(m2) the CNN for occupied class correspondence spot center detection” which is the learning model (CNN) generated on the basis of images of occupied parking spots has higher object similarity with (b1) the spot center estimation target parking spot (occupied parking spot).
Whereas, “(m1) the CNN for vacant class correspondence spot center detection” which is the learning model (CNN) generated on the basis of images of vacant parking spots has lower object similarity with (b1) the spot center estimation target parking spot (occupied parking spot), so that a heat map with a small peak is generated.
These two spot center identification heat maps having the different peaks are input to the parking spot state (vacant/occupied) determination unit 141 of the estimation result analysis unit 124 of the parking spot analysis unit 120 illustrated in
The parking spot state (vacant/occupied) determination unit 141 determines that the learning model (CNN) on a side that has output the spot center identification heat map having a large peak is close to a state of the parking spot as a parking spot state (vacant/occupied) determination target.
For example, in the example illustrated in
In this case, the parking lot as a parking spot state (vacant/occupied) determination target is determined to be an occupied parking spot in which a parked vehicle is present.
Note that this processing will be described again later.
As described with reference to
Moreover, as illustrated in
As illustrated
Next, processing executed by the spot center relative position estimation unit 132 in the parking spot configuration estimation unit 123 of the parking spot analysis unit 120 illustrated in
Processing executed by the spot center relative position estimation unit 132 will be described with reference to
As described above with reference to
However, the spot center grid estimation unit 131 merely estimates one grid including the center position of the parking spot. That is, an actual center position of the parking spot does not necessarily coincide with a center of the spot center grid.
The spot center relative position estimation unit 132 estimates the actual center position of the parking spot.
Specifically, as illustrated in
This processing will be described with reference to
“(1) The parking spot center grid estimation example” indicates a spot center grid estimated in the processing of the spot center grid estimation unit 131 described above with reference to
The actual spot center is within this spot center grid, but does not necessarily coincide with the grid center, and is often located at a position shifted from the grid center as shown in “(2) the parking spot center relative position estimation example”.
The actual spot center can be obtained by analyzing a peak position of the spot center identification heat map generated in the processing of the spot center grid estimation unit 131 described above with reference to
The spot center relative position estimation unit 132 analyzes the peak position of the spot center identification heat map on a pixel basis of the image instead of on a grid basis, and estimates an “actual parking spot center position” in the spot center grid as illustrated in
Moreover, as illustrated in
Next, a description is given to processing executed by the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134 in the parking spot configuration estimation unit 123 of the parking spot analysis unit 120 illustrated in
With reference to
First, with reference to
The spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134 have the same processing purpose, and the purpose thereof is, as illustrated in
“(1) The parking spot definition polygon four-vertex relative position” is a relative position (vector) of four vertexes of a polygon (rectangle) that defines a region of a parking spot from the actual spot center estimated by the spot center relative position estimation unit 132 described above with reference to
“(2) The parking spot entrance direction” is an entrance direction in a case of entering the parking spot.
“(1) The parking spot definition polygon four-vertex relative position” can be estimated on the basis of “CenterNet” which is a learning model applied to the spot center grid estimation processing that is performed by the spot center grid estimation unit 131 and is described above with reference to
As described above, the “CenterNet” is a learning model that enables estimation of a region of the entire object by analyzing center positions of various objects and calculating an offset from the center position to an end point of the object.
By applying the “CenterNet”, it is possible to calculate a spot center, and to estimate polygon vertexes of a parking spot, from a feature of the spot center and, for example, a feature amount detected by the feature amount detection unit 121, specifically, a white line that defines the parking lot, a vehicle stop block, a parked vehicle, and the like.
“(2) The parking spot entrance direction” is executed as processing of selecting two vertexes on the entrance side of the parking spot from the four polygon vertexes, after “(1) the parking spot definition polygon four-vertex relative position” is obtained.
As described above with reference to
By inputting a parking spot image as a processing target or feature amount data acquired from the parking spot image to such a learning model for analysis, it is possible to estimate a polygon vertex on the entrance side of the parking spot as a processing target.
As described above, the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134 have the same processing purpose, and estimates, as illustrated in
A difference between the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134 is an arrangement algorithm of vertexes of the parking spot definition polygon.
A difference between these two polygon vertex arrangement algorithms will be described with reference to
First, with reference to
These are a first vertex (x1, y1) to a fourth vertex (x4, y4) illustrated in
The spot vertex relative position and entrance estimation first algorithm execution unit 133 performs processing of arranging the four vertexes, that is, the first vertex (x1, y1) to the fourth vertex (x4, y4) of the parking spot definition four-vertex polygon 251 in accordance with the first algorithm.
As illustrated in an upper part of
A reference point 253 illustrated in
In accordance with the following first algorithm, namely, the vertex arrangement algorithm of
That is, a point at the upper left end closest to the reference point 253 is selected as the first vertex (x1, y1). Thereafter, the second vertex (x2, y2), the third vertex (x3, y3), and the fourth vertex (x4, y4) are sequentially selected clockwise.
As illustrated in
Next, with reference to
These are a second vertex (x1, y1) to a fourth vertex (x4, y4) illustrated in
The spot vertex relative position and entrance estimation second algorithm execution unit 134 performs processing of arranging the four vertexes, that is, the second vertex (x1, y1) to the fourth vertex (x4, y4) of the parking spot definition four-vertex polygon 251 in accordance with a second algorithm.
As illustrated in an upper part of
A parking spot image 250 illustrated in
In accordance with the second algorithm, namely, the following vertex arrangement algorithm:
That is, a point at the upper right end closest to the upper end of the image is selected as the first vertex (x1, y1). Thereafter, the second vertex (x2, y2), the third vertex (x3, y3), and the fourth vertex (x4, y4) are sequentially selected clockwise.
As illustrated in
In this way, in the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134, an arrangement algorithm of four vertexes, that is, the first vertex (x1, y1) to the fourth vertex (x4, y4), of the parking spot definition four-vertex polygon 251 is different.
The reason why the two types of vertex arrangement algorithms are executed in parallel in this manner is that a vertex arrangement error may occur if only one algorithm is used.
A specific example in which a vertex arrangement error occurs will be described with reference to
The first algorithm is a vertex arrangement algorithm of “among four vertexes constituting a parking spot definition four-vertex polygon, the closest point from a reference point (an upper left end point of a polygon circumscribed rectangle) is defined as a first vertex, and thereafter, a second, third, and fourth vertexes are defined clockwise”.
The parking spot definition four-vertex polygon 251 illustrated in
In a case of such a setting, both two points of vertexes P and Q of the parking spot definition polygon 251 illustrated in
As a result, in a case of selecting the first vertex in accordance with the first algorithm described above, there is a possibility that both of the points P and Q are selected as the first vertex (x1, y1), and a failure of the algorithm occurs.
In such a case, the first algorithm cannot be used.
Next, with reference to
The second algorithm is a vertex arrangement algorithm of “among four vertexes constituting a parking spot definition four-vertex polygon, the closest point from an upper end of the image is defined as a first vertex, and thereafter, a second, third, and fourth vertexes are defined clockwise”.
The parking spot definition four-vertex polygon 251 illustrated in
In a case of such a setting, both two points of vertexes R and S of the parking spot definition polygon 251 illustrated in
As a result, in a case of selecting the first vertex in accordance with the second algorithm described above, there is a possibility that both of the points R and S are selected as the first vertex (x1, y1), and a failure of the algorithm occurs.
In such a case, the second algorithm cannot be used.
In this way, in both the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134, the vertex arrangement cannot be performed in a case where the arrangement of the parking spot definition four-vertex polygon 251 is a specific arrangement.
In order to solve this problem, the information processing device according to the present disclosure has a configuration provided with two processing units in the parking spot configuration estimation unit 123 of the parking spot analysis unit 120, that is, the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134.
These two algorithm execution units execute the spot vertex relative position and entrance estimation processing in parallel.
Two “spot vertex relative position and entrance estimation results” as results of execution by these two algorithm execution units are output to the “spot vertex relative position and entrance estimation result selection unit 142” provided in the next estimation result analysis unit 124.
The “spot vertex relative position and entrance estimation result selection unit 142” of the estimation result analysis unit 124 selects one estimation result from the two estimation results input from the two algorithm execution units.
With reference to
As illustrated in
Moreover, the spot vertex relative position and entrance estimation result selection unit 142 is input with an estimation result of a spot vertex pattern from the spot vertex pattern estimation unit 135 of the parking spot configuration estimation unit 123 in the previous stage.
The spot vertex pattern estimation unit 135 of the parking spot configuration estimation unit 123 performs processing of estimating an inclination, a shape, and the like of the parking spot definition four-vertex polygon. This estimation processing is executed using a learning model.
Specifically, an inclination of the parking spot definition four-vertex polygon, that is, an inclination with respect to an input image (top surface image) and an inclination angle with respect to a circumscribed rectangle are analyzed, and an analysis result is input to the spot vertex relative position and entrance estimation result selection unit 142 of the estimation result analysis unit 124.
Note that the estimation processing in the spot vertex pattern estimation unit 135 is not limited to that using the learning model, and may be executed on a rule basis. In a case where the estimation processing is executed on a rule basis, a result obtained by analyzing, on a rule basis, an inclination of the parking spot definition four-vertex polygon, that is, an inclination with respect to the input image (top surface image), and an inclination angle with respect to the circumscribed rectangle may be input to the spot vertex relative position and entrance estimation result selection unit 142 of the estimation result analysis unit 124.
On the basis of inclination information of the parking spot definition four-vertex polygon input from the spot vertex pattern estimation unit 135, the spot vertex relative position and entrance estimation result selection unit 142 of the estimation result analysis unit 124 determines which estimation result to select from among the estimation result of the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the estimation result of the spot vertex relative position and entrance estimation second algorithm execution unit 134.
Specifically, for example, as described above with reference to
Furthermore, for example, as described above with reference to
In this way, in a case where there is a possibility that any of the algorithms will cause an error, without selecting the estimation result of the algorithm, processing of selecting an estimation result of another algorithm is performed.
By performing such processing, it becomes possible to select a correct estimation result of a spot vertex relative position and an entrance for all inclinations of the parking spot definition four-vertex polygon, and use the estimation result for the subsequent processing.
Next, with reference to
The parking spot state (vacant/occupied) determination unit 141 of the estimation result analysis unit 124 determines whether the parking lot is in a vacant state where no parked vehicle is present or in an occupied state where a parked vehicle is present.
As illustrated in
That is, the two heat maps are the following two spot center identification heat maps generated by the spot center grid estimation unit 131 of the parking spot configuration estimation unit 123.
These two heat maps are input from the spot center grid estimation unit 131 of the parking spot configuration estimation unit 123.
In the example illustrated in
This is due to similarity between a parking spot (object) as a spot center determination target in the spot center grid estimation unit 131 of the parking spot configuration estimation unit 123 and an object class of the used learning model (CNN) as described above.
That is, it means that the parking spot in the spot center estimation target image is a vacant parking spot.
In a case where the parking spot in the spot center estimation target image is a vacant parking spot, a peak (output value) of the heat map generated using the learning model (CNN) generated on the basis of images of vacant parking spots, namely,
The parking spot state (vacant/occupied) determination unit 141 determines that the learning model (CNN) on a side that has output the spot center identification heat map having a large peak is close to a state of the parking spot as a parking spot state (vacant/occupied) determination target.
For example, in the example illustrated in
In this case, the parking spot state (vacant/occupied) determination unit 141 determines that the determination target parking spot is a vacant parking spot in which no parked vehicle is present.
These two heat maps are input from the spot center grid estimation unit 131 of the parking spot configuration estimation unit 123.
In the example illustrated in
This is due to similarity between a parking spot (object) as a spot center determination target in the spot center grid estimation unit 131 of the parking spot configuration estimation unit 123 and an object class of the used learning model (CNN).
That is, it means that the parking spot in the spot center estimation target image is an occupied parking spot.
In a case where the parking spot in the spot center estimation target image is an occupied parking spot, a peak (output value) of the heat map generated using the learning model (CNN) generated on the basis of images of occupied parking spots, namely,
The parking spot state (vacant/occupied) determination unit 141 determines that the learning model (CNN) on a side that has output the spot center identification heat map having a large peak is close to a state of the parking spot as a parking spot state (vacant/occupied) determination target.
In the example illustrated in
In this case, the parking spot state (vacant/occupied) determination unit 141 determines that the determination target parking spot is an occupied parking spot in which parked vehicle is present.
Next, with reference to
In
The processing executed by the rescale unit 143 is step S101. the processing executed by the parking spot center coordinate calculation unit 144 is step S102, the processing executed by the parking spot definition polygon vertex coordinate calculation unit 145 is step S103, and the processing executed by the parking spot definition polygon coordinate rearrangement unit 146 is step S104.
Hereinafter, the processing of each step is described sequentially.
First, in step S101, the rescale unit 143 is input with an image used by the parking spot state (vacant/occupied) determination unit 141 for the parking spot state (vacant/occupied) determination processing, and executes rescale processing of matching the image with a resolution level of an original input image or a resolution level of an output image, that is, an output image to be output to the display unit of the vehicle 10.
For example, in a case where down-sampling is performed in the down-sampling unit 122 described above with reference to
Next, in step S102, the parking spot center coordinate calculation unit 144 executes adjustment processing on the parking spot center coordinates. That is, a coordinate position of the parking spot center coordinates corresponding to the resolution of the rescaled output image is calculated.
The parking spot center coordinate calculation unit 144 is input with spot center relative position information from the spot center relative position estimation unit 132 of the parking spot configuration estimation unit 123 in the previous stage.
This is the processing described above with reference to
However, this vector (offset) is calculated on the basis of the down-sampled data. Therefore, in step S102, the parking spot center coordinate calculation unit 144 performs adjustment processing on the parking spot center coordinates, that is, calculates the coordinate position of the parking spot center coordinates according to the resolution of the rescaled output image.
Next, in step S103, the parking spot definition polygon vertex calculation unit 145 executes adjustment processing on the parking spot definition polygon four-vertex coordinates. Specifically, the coordinate position, calculation, and the like according to the output image resolution are executed.
The parking spot definition polygon vertex calculation unit 145 is input with the spot vertex relative position and entrance estimation result from the spot vertex relative position and entrance estimation result selection unit 142 in the previous stage.
As described above, this is one error-free estimation result selected from the estimation results of the two algorithms estimated by the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134 in the parking spot configuration estimation unit 123 in the previous stage.
However, since the spot vertex relative position, the entrance position, and the like included in the estimation result are also calculated on the basis of the down-sampled data, in step S103, the parking spot definition polygon vertex calculation unit 145 executes adjustment processing on the parking spot definition polygon four-vertex coordinates. Specifically, the coordinate position, calculation, and the like according to the output image resolution are executed.
Finally, in step S104, the parking spot definition polygon coordinate rearrangement unit 146 executes processing of rearranging the polygon four-vertex coordinates corresponding to each parking spot, in accordance with an edge position on the entrance side of each parking spot.
The parking spot definition polygon coordinate rearrangement unit 146 is also input with the spot vertex relative position and entrance estimation result from the spot vertex relative position and entrance estimation result selection unit 142 in the previous stage.
That is, this is one error-free estimation result selected from the estimation results of the two algorithms estimated by the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134 in the parking spot configuration estimation unit 123 in the previous stage.
Information to be input from the spot vertex relative position and entrance estimation result selection unit 142 in the previous stage includes four-vertex information of the parking spot definition polygon and two-vertex information on the entrance edge side.
However, an arrangement order of the arrangement of the polygon four vertexes, that is, the first vertex (x1, y1) to the fourth vertex (x4, y4), which are the four vertexes of the parking spot definition polygon described above with reference to
The parking spot definition polygon coordinate rearrangement unit 146 performs processing of rearranging these irregular vertex arrangement to align the entrance directions of the parking spots.
That is, for example, as illustrated in
The parking spot definition polygon vertex arrangement data subjected to the rearrangement is input to a display control unit. For example, the display control unit can perform processing of displaying parking spot identification frames to be displayed with the first vertex and the fourth vertex arranged side by side on the entrance side, for all the adjacent parking spots.
As illustrated in
For example, in a case of a manual driving vehicle, the driver can reliably and easily determine a vacant or occupied state and an entrance direction of each parking frame, on the basis of the above-described identification data of (1) to (4) displayed on the display unit.
Furthermore, in a case of an automated driving vehicle, an image (top surface image) to which the identification data described above is added is input to the automated driving control unit. The automated driving control unit can reliably and easily determine a vacant or occupied state and an entrance direction of each parking frame on the basis of these pieces of identification data, and can perform automated parking processing accompanied by highly accurate position control for the vacant parking spot.
Next, other embodiments will be described.
In the above-described embodiment, the embodiment in which the image input to the parking spot analysis unit 120 is a top surface image has been described.
Namely, as described above with reference to
The vehicle 10 is equipped with these four cameras, combines images captured by these four cameras to generate an image observed from above, that is, a top surface image (overhead view image), and inputs the composite image to the parking spot analysis unit 120 to execute the parking spot analysis processing.
However, the image input to and analyzed by the parking spot analysis unit 120 is not limited to such a top surface image.
For example, as illustrated in
However, in this case, the parking spot analysis unit 120 executes analysis processing using a learning model generated using an image captured by one camera 11 that captures an image in the front direction of the vehicle 10.
Display data based on analysis data obtained by inputting an image captured by one camera 11 that captures an image in the front direction of the vehicle 10 to the parking spot analysis unit 120 and executing the parking spot analysis processing is to be display data as illustrated in
The display data of the display unit 12 illustrated in
Note that the identification data described above, namely,
The parking spot analysis unit 120 executes analysis processing using a learning model generated using an image captured in the vehicle front direction by one camera.
In this way, the information processing device according to the present disclosure can be used for the parking spot analysis processing using various images.
Next, a configuration example of the information processing device according to the present disclosure will be described.
As illustrated in
The parking spot analysis unit 120 includes the feature amount extraction unit 121, the down-sampling unit 122, the parking spot configuration estimation unit 123, and the estimation result analysis unit 124.
The display control unit 150 includes a parking spot state (vacant/occupied) identification frame generation unit 151, a parking spot entrance identification data generation unit 152, and a parking spot state (vacant/occupied) identification tag generation unit 153.
Note that the automated driving control unit 200 is not an essential configuration, and is a configuration included in a case where the vehicle is a vehicle that can perform automated driving.
The camera 101 includes, for example, multiple cameras that capture images in the front, rear, left, and right directions of the vehicle described with reference to
Note that, although not illustrated in
Note that the light detection and ranging (LiDAR) and the ToF sensor are, for example, a sensor that outputs light such as laser light, analyzes reflected light by an object, and measures a distance of a surrounding object.
As illustrated in the figure, an image captured by the camera 101 is input to the image conversion unit 102. For example, the image conversion unit 102 combines input images from multiple cameras that capture images in the vehicle front, rear, left, and right directions to generate a top surface image (overhead view image), and outputs the top surface image to the feature amount extraction unit 121 and the down-sampling unit 122 of the parking spot analysis unit 120.
Moreover, the top surface image (overhead view image) generated by the image conversion unit 102 is displayed on the display unit 1260 via the display control unit 150.
The parking spot analysis unit 120 includes the feature amount extraction unit 121, the down-sampling unit 122, the parking spot configuration estimation unit 123, and the estimation result analysis unit 124.
A configuration and processing of the parking spot analysis unit 120 are as described above with reference to
The feature amount extraction unit 121 extracts a feature amount from a top surface image that is an input image.
The feature amount extraction unit 121 executes the feature amount extraction processing using the learning model 180 generated by the learning processing unit 80 described above with reference to
The down-sampling unit 122 executes down-sampling processing on the feature amount data extracted from the input image (top surface image) by the feature amount extraction unit 121. Note that the down-sampling processing is performed to reduce a processing load in the parking spot configuration estimation unit 123, and is not essential.
The parking spot configuration estimation unit 123 is input with the input image (top surface image) and the feature amount data extracted from the image by the feature amount extraction unit 121, and executes analysis processing on a configuration, a state (vacant/occupied), and the like of the parking spot included in the input image.
The learning model 180 is also used in the parking spot analysis processing in the parking spot configuration estimation unit 123.
The learning model used by the parking spot configuration estimation unit 123 is, for example, learning models as follows:
As described above with reference to
The spot center grid estimation unit 131 estimates a spot center grid of each parking spot in the input image.
This processing is processing corresponding to the processing described above with reference to
That is, two learning models (CNN) are used. Namely, by inputting image data of a spot center estimation target parking spot or feature amount data on a grid basis acquired from the image data to two learning models (CNN), which are:
Namely, two heat maps of:
A spot center grid is estimated on the basis of peak positions of these heat maps.
The spot center relative position estimation unit 132 estimates an actual center position of the parking spot. Specifically, as described above with reference to
The spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134 select a relative position of parking spot definition polygon four vertexes and a parking spot entrance by using different algorithms.
Processing and an algorithm of these processing units are as described above with reference to
Note that the spot vertex pattern estimation unit 135 estimates an inclination, a shape, and the like of the parking spot definition four-vertex polygon. This estimation information is used to determine which of the two algorithms described above is to be selected.
As described above with reference to
The parking spot state (vacant/occupied) determination unit 141 determines a parking spot state, that is, whether the parking spot is a vacant spot without a parked vehicle or an occupied spot with a parked vehicle.
Specifically, as described above with reference to
The spot vertex relative position and entrance estimation result selection unit 142 selects one error-free estimation result from the spot vertex relative position and entrance estimation results according to individual algorithms and individually input from the spot vertex relative position and entrance estimation first algorithm execution unit 133 and the spot vertex relative position and entrance estimation second algorithm execution unit 134 in the previous stage.
This processing is the processing described above with reference to
The rescale unit 143, the parking spot center coordinate calculation unit 144, the parking spot definition polygon vertex coordinate calculation unit 145, and the parking spot definition polygon coordinate rearrangement unit 146 execute the processing described above with reference to
The display control unit 150 is input with an analysis result of the parking spot analysis unit 120, and executes generation processing for data to be displayed on the display unit 160 by using the input analysis result.
The display control unit 150 includes the parking spot state (vacant/occupied) identification frame generation unit 151, the parking spot entrance identification data generation unit 152, and the parking spot state (vacant/occupied) identification tag generation unit 153.
The parking spot state (vacant/occupied) identification frame generation unit 151 generates different identification frames according to a state (vacant/occupied) of the parking spot.
For example, a blue frame is used for a vacant spot, a red frame is used for an occupied spot, and the like.
The parking spot entrance identification data generation unit 152 generates identification data that enables identification of an entrance of each parking spot. For example, the identification data is the arrow data described with reference to
The parking spot state (vacant/occupied) identification tag generation unit 153 generates the identification tag corresponding to the parking spot state (vacant/occupied) as described above with reference to
The identification data generated by the display control unit 150 is displayed on the display unit 160 superimposed on the top surface image generated by the image conversion unit 102.
For example, as described above with reference to
The input unit (UI) 170 is a UI to be used for processing for inputting a parkable space search processing start instruction, processing for inputting target parking spot selection information, or the like, by a driver who is the user, for example. The input unit (UI) 170 may have a configuration using a touch panel formed on the display unit 160.
The input information of the input unit (UI) 170 is input to, for example, the automated driving control unit 200.
For example, the automated driving control unit 200 is input with analysis information of the parking spot analysis unit 120, display data generated by the display control unit 150, and the like, and executes automated driving processing toward a closest parking spot in a vacant state and automated parking processing.
Furthermore, for example, an automated parking processing for a parking spot designated according to designation information or the like of a target parking spot input from the input unit (UI) 170 is executed.
Next, a hardware configuration example of the information processing device according to the present disclosure will be described with reference to
Note that the information processing device is mounted in the vehicle 10. The hardware configuration illustrated in
The hardware configuration illustrated in
A central processing unit (CPU) 301 functions as a data processing unit configured to execute various types of processing in accordance with a program stored in a read only memory (ROM) 302 or a storage unit 308. For example, the CPU 301 executes the processing according to the sequence described in the above embodiment. A random access memory (RAM) 303 stores programs, data, or the like to be performed by the CPU 301. The CPU 301, the ROM 302, and the RAM 303 are connected to each other by a bus 304.
The CPU 301 is connected to an input/output interface 305 via the bus 304, and to the input/output interface 305, an input unit 306 that includes various switches, a touch panel, a microphone, and a status data acquisition unit of a user input unit and various sensors 321 such as a camera and LiDAR, and an output unit 307 that includes a display, a speaker, or the like are connected.
Furthermore, the output unit 307 also outputs drive information for a drive unit 322 of the vehicle.
The CPU 301 inputs commands, status data, or the like input from the input unit 306, executes various types of processing, and outputs processing results to, for example, the output unit 307.
The storage unit 308 connected to the input/output interface 305 includes, for example, a hard disk, or the like and stores programs executed by the CPU 301 and various types of data. A communication unit 309 functions as a transmission/reception unit for data communication via a network such as the Internet or a local area network, and communicates with an external device.
Furthermore, in addition to the CPU, a graphics processing unit (GPU) may be provided as a dedicated processing unit for image information and the like input from the camera.
A drive 310 connected to the input/output interface 305 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and records or reads data.
Next, a configuration example of a vehicle on which the information processing device according to the present disclosure is mounted will be described.
The vehicle control system 511 is provided in the vehicle 500 and executes processing related to travel assistance and automated driving of the vehicle 500.
The vehicle control system 511 includes a vehicle control electronic control unit (ECU) 521, a communication unit 522, a map information accumulation unit 523, a global navigation satellite system (GNSS) reception unit 524, an external recognition sensor 525, an in-vehicle sensor 526, a vehicle sensor 527, a recording unit 528, a travel assistance/automated driving control unit 529, a driver monitoring system (DMS) 530, a human machine interface (HMI) 531, and a vehicle control unit 532.
The vehicle control electronic control unit (ECU) 521, the communication unit 522, the map information accumulation unit 523, a GNSS reception unit 524, the external recognition sensor 525, the in-vehicle sensor 526, the vehicle sensor 527, the recording unit 528, the travel assistance/automated driving control unit 529, the driver monitoring system (DMS) 530, the human machine interface (HMI) 531, and the vehicle control unit 532 are communicably connected to each other via a communication network 41. A communication network 241 includes, for example, an in-vehicle communication network, a bus, or the like that conforms to a digital bidirectional communication standard, such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), or Ethernet (registered trademark). The communication network 241 may be selectively used depending on the type of data to be communicated, and for example, the CAN is applied to data related to vehicle control, and the Ethernet is applied to large-capacity data. Note that units of the vehicle control system 511 may be directly connected to each other using wireless communication adapted to a relatively short-range communication, such as near field communication (NFC) or Bluetooth (registered trademark) without using the communication network 241, for example.
Note that, hereinafter, in a case where each unit of the vehicle control system 511 performs communication via the communication network 241, the description of the communication network 241 will be omitted. For example, in a case where the vehicle control electronic control unit (ECU) 521 and the communication unit 522 perform communication via the communication network 241, it is simply described that a processor and the communication unit 522 perform communication.
The vehicle control electronic control unit (ECU) 521 includes, for example, various processors such as a central processing unit (CPU) or a micro processing unit (MPU). The vehicle control electronic control unit (ECU) 521 controls the entire of partial function of the vehicle control system 511.
The communication unit 522 communicates with various devices inside and outside the vehicle, another vehicle, a server, a base station, and the like, and transmits and receives various types of data. At this time, the communication unit 522 can perform communication using a plurality of communication schemes.
Communication with the outside of the vehicle executable by the communication unit 522 will be schematically described. The communication unit 522 communicates with a server (hereinafter, referred to as an external server) or the like that exists on an external network via a base station or an access point by, for example, a wireless communication scheme such as fifth generation mobile communication system (5G), long term evolution (LTE), dedicated short range communications (DSRC), or the like. Examples of the external network with which the communication unit 522 performs communication include the Internet, a cloud network, a company-specific network, or the like. The communication method by which the communication unit 522 communicates with the external network is not particularly limited as long as it is a wireless communication method capable of performing digital bidirectional communication at a communication speed equal to or more than a predetermined speed and at a distance equal to or longer than a predetermined distance.
Furthermore, for example, the communication unit 522 can communicate with a terminal present in the vicinity of the own vehicle using a peer to peer (P2P) technology. The terminal present in the vicinity of the own vehicle is, for example, a terminal worn by a moving body moving at a relatively low speed such as a pedestrian or a bicycle, a terminal installed in a store or the like with a position fixed, or a machine type communication (MTC) terminal. Moreover, the communication unit 522 can also perform V2X communication. The V2X communication refers to, for example, communication between the own vehicle and another vehicle, such as vehicle to vehicle communication with another vehicle, vehicle to infrastructure communication with a roadside device or the like, vehicle to home communication, and vehicle to pedestrian communication with a terminal or the like carried by a pedestrian.
For example, the communication unit 522 can receive a program for updating software for controlling the operation of the vehicle control system 511 from the outside (Over The Air). The communication unit 522 can further receive map information, traffic information, the information regarding the surroundings of the vehicle 500, or the like from the outside. Furthermore, for example, the communication unit 522 can transmit information regarding the vehicle 500, information regarding the surroundings of the vehicle 500, or the like to the outside. Examples of the information regarding the vehicle 500 transmitted to the outside by the communication unit 522 include data indicating a state of the vehicle 500, a recognition result from a recognition unit 573, or the like. Moreover, for example, the communication unit 522 performs communication corresponding to a vehicle emergency call system such as an eCall.
Communication with the inside of the vehicle executable by the communication unit 522 will be schematically described. The communication unit 522 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 522 can perform wireless communication with the device in the vehicle by, for example, a communication scheme allowing digital bidirectional communication at a communication speed equal to or higher than a predetermined speed by wireless communication, such as wireless LAN, Bluetooth, NFC, or wireless USB (WUSB). The communication performed by the communication unit 522 is not limited to this, and the communication unit 522 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 522 can communicate with each device in the vehicle by wired communication via a cable connected to a connection terminal (not illustrated). The communication unit 522 can communicate with each device in the vehicle by a communication scheme allowing digital bidirectional communication at a communication speed equal to or higher than a predetermined speed by wired communication, for example, a universal serial bus (USB), the high-definition multimedia interface (HDMI) (registered trademark), a mobile high-definition link (MHL), or the like.
Here, the device in the vehicle refers to, for example, a device that is not connected to the communication network 241 in the vehicle. As the device in the vehicle, for example, a mobile device or a wearable device carried by an occupant such as a driver, an information device brought into the vehicle and temporarily installed, or the like is assumed.
For example, the communication unit 522 receives an electromagnetic wave transmitted by a road traffic information communication system (vehicle information and communication system (VICS) (registered trademark)), such as a radio wave beacon, an optical beacon, or FM multiplex broadcasting.
The map information accumulation unit 523 accumulates one or both of a map acquired from the outside and a map created by the vehicle 500. For example, the map information accumulation unit 523 accumulates a three-dimensional high-precision map, a global map that is lower in precision than the high-precision map but covers a wider area, and the like.
The high-precision map is, for example, a dynamic map, a point cloud map, a vector map, or the like. The dynamic map is, for example, a map including four layers of dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to the vehicle 500 from the external server or the like. The point cloud map is a map including a point cloud (point cloud data). Here, the vector map indicates a map adapted to an advanced driver assistance system (ADAS) in which traffic information such as a lane and a signal position is associated with the point cloud map.
The point cloud map and the vector map may be provided from, for example, an external server or the like, or may be created by the vehicle 500 as a map for performing matching with a local map to be described later on the basis of a sensing result by a radar 552, a LiDAR 553, or the like, and may be accumulated in the map information accumulation unit 523.
Furthermore, in a case where the high-precision map is provided from the external server or the like, for example, map data of several hundred meters square regarding a planned path on which the vehicle 500 travels from now is acquired from the external server or the like in order to reduce the communication traffic.
The GNSS reception unit 524 receives a GNSS signal from a GNSS satellite and acquires position information of the vehicle 500. The received GNSS signal is supplied to the travel assistance/automated driving control unit 529. Note that the GNSS reception unit 524 may acquire the position information, for example, using a beacon, without limiting to the method using the GNSS signal.
The external recognition sensor 525 includes various sensors used to recognize a situation outside the vehicle 500, and supplies sensor data from each sensor to each unit of the vehicle control system 511. The type and number of sensors included in the external recognition sensor 525 may be determined as desired.
For example, the external recognition sensor 525 includes a camera 551, the radar 552, the light detection and ranging, laser imaging detection and ranging (LiDAR) 553, and an ultrasonic sensor 554. Without being limited to this, and the external recognition sensor 525 may include one or more types of sensors among the camera 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554. The numbers of the cameras 551, the radars 552, the LiDARs 553, and the ultrasonic sensors 554 are not particularly limited as long as the sensors can be provided in the vehicle 500. Furthermore, the external recognition sensor 525 may include sensors of other types, but not limited to sensors of the types described in this example. An example of a sensing region of each sensor included in the external recognition sensor 525 will be described later.
Note that the imaging method of the camera 551 is not particularly limited as long as it is an imaging method capable of distance measurement. For example, as the camera 551, cameras of various imaging methods such as a time of flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera can be applied as necessary. Without being limited to this, and the camera 551 may simply acquire a captured image regardless of distance measurement.
Furthermore, for example, the external recognition sensor 525 can include an environment sensor for detecting an environment for the vehicle 500. The environmental sensor is a sensor for detecting an environment such as weather, climate, and brightness and can include various sensors such as a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, and an illuminance sensor, for example.
Moreover, for example, the external recognition sensor 525 includes a microphone used to detect a sound around the vehicle 500, a position of a sound source, or the like.
The in-vehicle sensor 526 includes various sensors for detecting information regarding the inside of the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 511. The types and the number of various sensors included in the in-vehicle sensor 526 are not particularly limited as long as they can be practically installed in the vehicle 500.
For example, the in-vehicle sensor 526 can include one or more sensors of a camera, a radar, a seating sensor, a steering wheel sensor, a microphone, and a biological sensor. As the camera included in the in-vehicle sensor 526, for example, cameras of various imaging methods capable of measuring a distance, such as a ToF camera, a stereo camera, a monocular camera, and an infrared camera, can be used. Without being limited to this, the camera included in the in-vehicle sensor 526 may simply acquire a captured image regardless of distance measurement. The biological sensor included in the in-vehicle sensor 526 is provided in, for example, a seat, a steering wheel, or the like, and detects various types of biological information of the occupant such as the driver.
The vehicle sensor 527 includes various sensors for detecting the state of the vehicle 500, and supplies the sensor data from each sensor to each unit of the vehicle control system 511. The types and the number of various sensors included in the vehicle sensor 527 are not particularly limited as long as they can be practically installed in the vehicle 500.
For example, the vehicle sensor 527 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) in which these sensors are integrated. For example, the vehicle sensor 527 includes a steering angle sensor that detects a steering angle of a steering wheel, a yaw rate sensor, an accelerator sensor that detects an operation amount of an accelerator pedal, and a brake sensor that detects an operation amount of a brake pedal. For example, the vehicle sensor 527 includes a rotation sensor that detects the number of rotations of an engine or a motor, an air pressure sensor that detects an air pressure of a tire, a slip rate sensor that detects a slip rate of the tire, and a wheel speed sensor that detects a rotation speed of a wheel. For example, the vehicle sensor 527 includes a battery sensor that detects a remaining amount and temperature of a battery, and an impact sensor that detects an external impact.
The recording unit 528 includes at least one of a non-volatile storage medium or a volatile storage medium, and stores data and a program. The recording unit 528 is used as, for example, an electrically erasable programmable read only memory (EEPROM) and a random access memory (RAM), and a magnetic storage device such as a hard disc drive (HDD), a semiconductor storage device, an optical storage device, and a magneto-optical storage device can be applied as the storage medium. The recording unit 528 records various programs and data used by each unit of the vehicle control system 511. For example, the recording unit 528 includes an event data recorder (EDR) and a data storage system for automated driving (DSSAD), and records information of the vehicle 500 before and after an event such as an accident and biological information acquired by the in-vehicle sensor 526.
The travel assistance/automated driving control unit 529 controls travel assistance and automated driving of the vehicle 500. For example, the travel assistance/automated driving control unit 529 includes an analysis unit 561, an action planning unit 562, and an operation control unit 563.
The analysis unit 561 executes analysis processing on the vehicle 500 and a situation around the vehicle 500. The analysis unit 561 includes a self-position estimation unit 571, a sensor fusion unit 572, and the recognition unit 573.
The self-position estimation unit 571 estimates a self-position of the vehicle 500, on the basis of the sensor data from the external recognition sensor 525 and the high-precision map accumulated in the map information accumulation unit 523. For example, the self-position estimation unit 571 generates a local map on the basis of the sensor data from the external recognition sensor 525 and performs matching the local map with the high-precision map so as to estimate the self-position of the vehicle 500. The position of the vehicle 500 is based on, for example, a center of a rear wheel pair axle.
The local map is, for example, a three-dimensional high-precision map created using a technology such as simultaneous localization and mapping (SLAM), an occupancy grid map, or the like. The three-dimensional high-precision map is, for example, the above-described point cloud map or the like. The occupancy grid map is a map in which a three-dimensional or two-dimensional space around the vehicle 500 is divided into grids (lattices) with a predetermined size, and an occupancy state of an object is represented on a grid basis. The occupancy state of the object is represented by, for example, presence or absence or an existence probability of the object. The local map is also used for detection processing and recognition processing on the situation outside the vehicle 500 by the recognition unit 573, for example.
Note that the self-position estimation unit 571 may estimate the self-position of the vehicle 500 on the basis of the GNSS signal and the sensor data from the vehicle sensor 527.
The sensor fusion unit 572 executes sensor fusion processing for combining a plurality of different types of sensor data (for example, image data supplied from the camera 551 and sensor data supplied from the radar 552), to acquire new information. Methods for combining different types of sensor data include integration, fusion, correspondence, or the like.
The recognition unit 573 executes the detection processing for detecting a situation outside the vehicle 500 and the recognition processing for recognizing a situation outside the vehicle 500.
For example, the recognition unit 573 executes the detection processing and the recognition processing on the situation outside the vehicle 500, on the basis of the information from the external recognition sensor 525, the information from the self-position estimation unit 571, the information from the sensor fusion unit 572, or the like.
Specifically, for example, the recognition unit 573 executes the detection processing, the recognition processing, or the like on the object around the vehicle 500. The object detection processing is, for example, processing for detecting presence or absence, size, shape, position, motion, or the like of an object. The object recognition processing is, for example, processing for recognizing an attribute such as a type of an object or identifying a specific object. The detection processing and the recognition processing, however, are not necessarily clearly separated and may overlap.
For example, the recognition unit 573 detects an object around the vehicle 500 by performing clustering to classify a point cloud based on the sensor data by the LiDAR 553, the radar 552, or the like for each cluster of a point cloud. As a result, the presence or absence, size, shape, and position of the object around the vehicle 500 are detected.
For example, the recognition unit 573 detects a motion of the object around the vehicle 500 by performing tracking for following a motion of the cluster of the point cloud classified by clustering. As a result, a speed and a traveling direction (movement vector) of the object around the vehicle 500 are detected.
For example, the recognition unit 573 detects or recognizes a vehicle, a person, a bicycle, an obstacle, a structure, a road, a traffic light, a traffic sign, a road sign, and the like with respect to the image data supplied from the camera 551. Furthermore, the type of the object around the vehicle 500 may be recognized by executing recognition processing such as semantic segmentation.
For example, the recognition unit 573 can execute processing for recognizing traffic rules around the vehicle 500 on the basis of the map accumulated in the map information accumulation unit 523, the estimation result of the self-position by the self-position estimation unit 571, and the recognition result of the object around the vehicle 500 by the recognition unit 573. Through this processing, the recognition unit 573 can recognize a position and state of a signal, content of traffic signs and road signs, content of traffic regulations, travelable lanes, and the like.
For example, the recognition unit 573 can execute the recognition processing on a surrounding environment of the vehicle 500. As the surrounding environment to be recognized by the recognition unit 573, a weather, a temperature, a humidity, brightness, a road surface condition, or the like are assumed.
The action planning unit 562 creates an action plan for the vehicle 500. For example, the action planning unit 562 creates the action plan by executing processing of path planning and path following.
Note that the route planning (global path planning) is processing of planning a rough route from a start to a goal. This path planning is called track planning, and also includes processing of track generation (local path planning) that allows safe and smooth traveling near the vehicle 500, in consideration of motion characteristics of the vehicle 500 in the path planned by the path planning. The path planning may be distinguished from long-term path planning, and startup generation from short-term path planning or local path planning. A safety-first path represents a concept similar to the startup generation, the short-term path planning, or the local path planning.
The path following is processing for planning an operation for safely and accurately traveling on the path planned by the path planning within a planned time. For example, the action planning unit 562 can calculate a target speed and a target angular velocity of the vehicle 500, on the basis of a result of the path following processing.
The operation control unit 563 controls the operation of the vehicle 500 in order to achieve the action plan created by the action planning unit 562.
For example, the operation control unit 563 controls a steering control unit 581, a brake control unit 582, and a drive control unit 583 included in the vehicle control unit 532 to be described later, to control acceleration/deceleration and the direction so that the vehicle 500 travels on a track calculated by the track planning. For example, the operation control unit 563 performs cooperative control for the purpose of implementing functions of the ADAS such as collision avoidance or impact mitigation, follow-up traveling, vehicle speed maintaining traveling, collision warning of the own vehicle, or lane deviation warning of the own vehicle. For example, the operation control unit 563 performs cooperative control for the purpose of automated driving or the like in which a vehicle autonomously travels without depending on an operation of a driver.
The DMS 530 executes authentication processing on the driver, recognition processing on a state of the driver, or the like, on the basis of the sensor data from the in-vehicle sensor 526, the input data input to the HMI 531 to be described later, or the like. In this case, as the state of the driver to be recognized by the DMS 530, for example, a physical condition, an alertness, a concentration degree, a fatigue degree, a line-of-sight direction, a degree of drunkenness, a driving operation, a posture, or the like are assumed.
Note that the DMS 530 may execute processing for authenticating an occupant other than the driver, and processing for recognizing a state of the occupant. Furthermore, for example, the DMS 530 may execute processing for recognizing a situation in the vehicle, on the basis of the sensor data from the in-vehicle sensor 526. As the situation in the vehicle to be recognized, for example, a temperature, a humidity, brightness, odor, or the like are assumed.
The HMI 531 receives inputs of various types of data, instructions, or the like, and presents various types of data to the driver or the like.
The input of data by the HMI 531 will be schematically described. The HMI 531 includes an input device for a person to input data. The HMI 531 generates an input signal on the basis of the data, the instruction, or the like input with the input device, and supplies the input signal to each unit of the vehicle control system 511. The HMI 531 includes, for example, an operator such as a touch panel, a button, a switch, or a lever as the input device. Without being limited to this, the HMI 531 may further include an input device capable of inputting information by a method such as voice or gesture other than a manual operation. Moreover, the HMI 531 may use, for example, a remote control device using infrared rays or radio waves, or an external connection device such as a mobile device or a wearable device corresponding to the operation of the vehicle control system 511, as the input device.
Presentation of data by the HMI 531 will be schematically described. The HMI 531 generates visual information, auditory information, and haptic information regarding an occupant or outside of a vehicle. Furthermore, the HMI 531 performs output control for controlling output, output content, an output timing, an output method, or the like of each piece of generated information. The HMI 531 generates and outputs, for example, information indicated by an image or light of an operation screen, a state display of the vehicle 500, a warning display, a monitor image indicating a situation around the vehicle 500, or the like, as the visual information. Furthermore, the HMI 531 generates and outputs information indicated by sounds such as voice guidance, a warning sound, or a warning message, for example, as the auditory information. Moreover, the HMI 531 generates and outputs, for example, information given to a tactile sense of an occupant by force, vibration, motion, or the like as the haptic information.
As an output device with which the HMI 531 outputs the visual information, for example, a display device that presents the visual information by displaying an image by itself or a projector device that presents the visual information by projecting an image can be applied. Note that the display device may be a device that displays the visual information in the field of view of the occupant, such as a head-up display, a transmissive display, or a wearable device having an augmented reality (AR) function, for example, in addition to a display device having an ordinary display. Furthermore, the HMI 531 can use a display device included in a navigation device, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like provided in the vehicle 500, as the output device that outputs the visual information.
As an output device with which the HMI 531 outputs the auditory information, for example, an audio speaker, a headphone, or an earphone can be applied.
As an output device with which the HMI 531 outputs the haptic information, for example, a haptic element using a haptic technology can be applied. The haptic element is provided, for example, in a portion to be touched by the occupant of the vehicle 500, such as a steering wheel or a seat.
The vehicle control unit 532 controls each unit of the vehicle 500. The vehicle control unit 532 includes the steering control unit 581, the brake control unit 582, the drive control unit 583, a body system control unit 584, a light control unit 585, and a horn control unit 586.
The steering control unit 581 performs detection, control, or the like of a state of a steering system of the vehicle 500. The steering system includes, for example, a steering mechanism including a steering wheel or the like, an electric power steering, or the like. The steering control unit 581 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, or the like.
The brake control unit 582 performs detection, control, or the like of a state of a brake system of the vehicle 500. The brake system includes, for example, a brake mechanism including a brake pedal or the like, an antilock brake system (ABS), a regenerative brake mechanism, or the like. The brake control unit 582 includes, for example, a control unit such as an ECU that controls the brake system, or the like.
The drive control unit 583 performs detection, control, or the like of a state of a drive system of the vehicle 500. The drive system includes, for example, an accelerator pedal, a driving force generation device for generating a driving force such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to wheels, or the like. The drive control unit 583 includes, for example, a control unit such as an ECU that controls the drive system, or the like.
The body system control unit 584 performs detection, control, or the like of a state of a body system of the vehicle 500. The body system includes, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioner, an airbag, a seat belt, a shift lever, or the like. The body system control unit 584 includes, for example, a control unit such as an ECU that controls the body system, or the like.
The light control unit 585 performs detection, control, or the like of states of various lights of the vehicle 500. As the lights to be controlled, for example, a headlight, a backlight, a fog light, a turn signal, a brake light, a projection light, a bumper indicator, or the like can be considered. The light control unit 585 includes a control unit such as an ECU that performs light control, or the like.
The horn control unit 586 performs detection, control, or the like of a state of a car horn of the vehicle 500. The horn control unit 586 includes, for example, a control unit such as an ECU that controls the car horn, or the like.
Sensing regions 591F and 591B illustrate examples of the sensing region of the ultrasonic sensor 554. The sensing region 591F covers a region around the front end of the vehicle 500 by the plurality of ultrasonic sensors 554. The sensing region 591B covers a region around the rear end of the vehicle 500 by the plurality of ultrasonic sensors 554.
Sensing results in the sensing regions 591F and 591B are used, for example, for parking assistance of the vehicle 500 or the like.
Sensing regions 592F to 592B illustrate examples of the sensing region of the radar 552 for short distance or medium distance. The sensing region 592F covers a position farther than the sensing region 591F, on the front side of the vehicle 500.
The sensing region 592B covers a position farther than the sensing region 591B, on the rear side of the vehicle 500. The sensing region 592L covers a region around the rear side of a left side surface of the vehicle 500. The sensing region 592R covers a region around the rear side of a right side surface of the vehicle 500.
A sensing result in the sensing region 592F is used for, for example, detection of a vehicle, a pedestrian, or the like existing on the front side of the vehicle 500, or the like. A sensing result in the sensing region 592B is used for, for example, a function for preventing a collision of the rear side of the vehicle 500, or the like. The sensing results in the sensing regions 592L and 592R are used for, for example, detection of an object in a blind spot on the sides of the vehicle 500, or the like.
Sensing regions 593F to 593B illustrate examples of the sensing regions by the camera 551. The sensing region 593F covers a position farther than the sensing region 592F, on the front side of the vehicle 500. The sensing region 593B covers a position farther than the sensing region 592B, on the rear side of the vehicle 500. The sensing region 593L covers a region around the left side surface of the vehicle 500. The sensing region 593R covers a region around the right side surface of the vehicle 500.
A sensing result in the sensing region 593F can be used for, for example, recognition of a traffic light or a traffic sign, a lane departure prevention assist system, and an automated headlight control system. A sensing result in the sensing region 593B can be used for, for example, parking assistance, a surround view system, or the like. Sensing results in the sensing regions 593L and 593R can be used for, for example, a surround view system.
A sensing region 594 illustrates an example of the sensing region of the LiDAR 553. The sensing region 594 covers a position farther than the sensing region 593F, on the front side of the vehicle 500. On the other hand, the sensing region 594 has a narrower range in a left-right direction than the sensing region 593F.
A sensing result in the sensing region 594 is used for, for example, detection of an object such as a neighboring vehicle.
A sensing region 595 illustrates an example of the sensing region of the long-distance radar 552.
The sensing region 595 covers a position farther than the sensing region 594, on the front side of the vehicle 500. On the other hand, the sensing region 595 has a narrower range in the left-right direction than the sensing region 594.
A sensing result in the sensing region 595 is used, for example, for adaptive cruise control (ACC), emergency braking, collision avoidance, or the like.
Note that the respective sensing regions of the individual sensors: the camera 551; the radar 552; the LiDAR 553; and the ultrasonic sensor 554, included in the external recognition sensor 525 may have various configurations other than those in
Furthermore, an installation position of each sensor is not limited to each example described above. Furthermore, the number of sensors of each sensor may be one or more.
Hereinabove, the embodiments according to the present disclosure have been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. That is, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present disclosure, the claims should be considered.
Note that the technology disclosed herein can have the following configurations.
(1) An information processing device including:
(2) The information processing device according to (1), in which
(3) The information processing device according to (1) or (2), in which
(4) The information processing device according to any one of (1) to (3), in which
(5) The information processing device according to (4), in which
(6) The information processing device according to (4) or (5), in which
(7) The information processing device according to any one of (1) to (6), in which
(8) The information processing device according to (7), in which
(9) The information processing device according to any one of (1) to (8), in which
(10) The information processing device according to (9), in which
(11) The information processing device according to (10), in which
(12) The information processing device according to (11), in which
(13) The information processing device according to (12), in which
(14) The information processing device according to (13), in which
(15) The information processing device according to any one of (1) to (14), in which the image is a top surface image corresponding to an image observed from above a vehicle, the top surface image being generated by combining images captured by four cameras that individually capture images in four directions of front, rear, left, and right, the four cameras being mounted on the vehicle.
(16) The information processing device according to any one of (1) to (15), in which
(17) The information processing device according to (16), in which
(18) The information processing device according to any one of (1) to (17), in which
(19) An information processing method executed in an information processing device, in which
(20) A program for causing an information processing device to execute information processing, in which
Furthermore, a series of processes described herein can be executed by hardware, software, or a configuration obtained by combining hardware and software. In a case of processing by software is executed, a program in which a processing sequence is recorded can be installed and performed in a memory in a computer incorporated in dedicated hardware, or the program can be installed and performed in a general-purpose computer capable of executing various types of processing. For example, the program can be recorded in advance in a recording medium. In addition to being installed in a computer from the recording medium, a program can be received via a network such as a local area network (LAN) or the Internet and installed in a recording medium such as an internal hard disk or the like.
Note that the various processes described herein may be executed not only in a chronological order in accordance with the description, but may also be executed in parallel or individually depending on processing capability of a device configured to execute the processing or depending on the necessity. Furthermore, a system herein described is a logical set configuration of a plurality of devices, there is a case where devices of each configuration are housed in the same housing. However, the system is not limited to a system in which devices of each configuration are in the same housing.
As described above, according to the configuration of one embodiment of the present disclosure, a configuration of estimating a parking spot definition rectangle (polygon), a parking spot entrance direction, and a vacancy state of a parking spot by applying a learning model is realized.
Specifically, for example, a top surface image generated by combining individual images captured by front, rear, left, and right cameras mounted on a vehicle is analyzed, and analysis processing is performed on a parking spot in the image. The parking spot analysis unit uses a learning model to estimate a vertex of a parking spot definition rectangle (polygon) indicating a parking spot region in the image and an entrance direction of the parking spot. Furthermore, estimation is performed as to whether the parking spot is a vacant parking spot or an occupied parking spot in which a parked vehicle is present. The parking spot analysis unit uses CenterNet as a learning model to execute, for example, estimation processing on a spot center and a vertex of a parking spot definition rectangle (polygon).
With this configuration, a configuration is realized in which a learning model is applied to estimate a parking spot definition rectangle (polygon), a parking spot entrance direction, and a vacancy state of a parking spot.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021-188329 | Nov 2021 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2022/038180 | 10/13/2022 | WO |
