Patent Application
20250216865
The present invention relates to a mobile object control device, a mobile object control method, and a storage medium.
Conventionally, practical applications have been made for mobile objects capable of moving on both a sidewalk and a roadway and it is necessary to set different upper limit speeds for the sidewalk and the roadway for such mobile objects. In this regard, documents considering a process of recognizing whether a mobile object is moving on a sidewalk or a roadway have been disclosed (Patent Document 1).
Japanese Unexamined Patent Application, First Publication No. 2020-086995
In the conventional technology, it may not be possible to appropriately recognize whether a mobile object is moving on a roadway or in a predetermined region different from the roadway.
The present invention has been made in consideration of such circumstances and an objective of the present invention is to provide a mobile object control device, a mobile object control method, and a storage medium for enabling whether a mobile object is moving on a roadway or in a predetermined region different from the roadway to be appropriately recognized.
A mobile object control device, a mobile object control method, and a storage medium according to the present invention adopt the following configurations.
According to the above-described aspects (1) to (14), it is possible to appropriately recognize whether a mobile object is moving on a roadway or in a predetermined region different from the roadway.
Hereinafter, embodiments of a mobile object control device, a mobile object control method, and a program according to the present invention will be described with reference to the drawings. A mobile object moves both on a roadway and in a predetermined region different from the roadway. The mobile object may be referred to as micromobility. An electric kickboard is a type of micromobility. Moreover, the mobile object may be a vehicle that an occupant can board or may be an autonomous mobile object capable of performing unmanned autonomous movement. The autonomous mobile object is used, for example, for an application to transport cargo or the like. The predetermined region is, for example, a sidewalk. Moreover, the predetermined region may be a part or all of a roadside strip, a bicycle lane, a public open space, or the like or may include all sidewalks, roadside strips, bicycle lanes, public open spaces, and the like. In the following description, it is assumed that the predetermined region includes a sidewalk and a public open space.
The external environment detection device 10 includes various types of devices in which a movement direction of the mobile object 1 is designated as a detection range. The external environment detection device 10 includes an external camera, a radar device, a light detection and ranging (LIDAR) sensor, a sensor fusion device, and the like. The external environment detection device 10 outputs information indicating a detection result (an image, a position of a physical object, or the like) to the control device 100.
The mobile object sensor 12 includes, for example, a speed sensor, an acceleration sensor, a yaw rate (angular velocity) sensor, an orientation sensor, a manipulation quantity detection sensor attached to the manipulation element 14, and the like. The manipulation element 14 includes, for example, a manipulation element for issuing an acceleration/deceleration instruction (for example, an accelerator pedal or a brake pedal) and a manipulation element for issuing a steering instruction (for example, a steering wheel). In this case, the mobile object sensor 12 may include an accelerator opening degree sensor, a brake depression amount sensor, a steering torque sensor, and the like. The mobile object 1 may include a type of manipulation element other than the above manipulation elements (for example, a non-annular rotation manipulation element, a joystick, a button, or the like) as the manipulation element 14.
The internal camera 16 images at least a head of the occupant of the mobile object 1 from the front. The internal camera 16 is a digital camera that uses an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The internal camera 16 outputs a captured image to the control device 100.
The positioning device 18 is a device for measuring a position of the mobile object 1. The positioning device 18 is, for example, a global navigation satellite system (GNSS) receiver, and identifies the position of the mobile object 1 on the basis of a signal received from a GNSS satellite and outputs the identified position as position information. Also, the position information of the mobile object 1 may be estimated from a position of a Wi-Fi base station to which a communication device to be described below is connected.
The mode changeover switch 22 is a switch to be manipulated by the occupant. The mode changeover switch 22 may be a mechanical switch or a graphical user interface (GUI) switch set on the touch panel. The mode changeover switch 22 receives a manipulation for switching a driving mode to any one of, for example, mode A that is an assist mode in which one of the steering manipulation and acceleration/deceleration control is performed by the occupant and the other is performed automatically and that may be mode A-1 in which the steering manipulation is performed by the occupant and the acceleration/deceleration control is performed automatically or mode A-2 in which an acceleration/deceleration manipulation is performed by the occupant and the steering control is performed automatically, mode B that is a manual driving mode in which the steering manipulation and the acceleration/deceleration manipulation are performed by the occupant, and mode C that is an automated driving mode in which the steering control and the acceleration/deceleration control are performed automatically.
The moving mechanism 30 is a mechanism for moving the mobile object 1 on the road. The moving mechanism 30 is, for example, a wheel group that includes a steered wheel and a driven wheel. Moreover, the moving mechanism 30 may be a leg unit for multi-legged walking.
The drive device 40 outputs a force to the moving mechanism 30 so that the mobile object 1 can be moved. For example, the drive device 40 includes a motor that drives the driven wheel, a battery that stores electric power to be supplied to the motor, a steering device that adjusts the steering angle of the steered wheel, and the like. The drive device 40 may include an internal combustion engine, a fuel cell, or the like as a driving force output means or an electric power generation means. Moreover, the drive device 40 may further include a braking device using a friction force or air resistance.
The external notification device 50 is, for example, a lamp, a display device, a speaker, or the like provided on an outer plate portion of the mobile object 1 and configured to provide a notification of information to the outside of the mobile object 1. The external notification device 50 performs different operations in a state in which the mobile object 1 is moving in a predetermined region and a state in which the mobile object 1 is moving on the roadway. For example, the external notification device 50 is controlled so that light of the lamp is emitted when the mobile object 1 is moving in the predetermined region and light of the lamp is not emitted when the mobile object 1 is moving on the roadway. The color of the light emitted by this lamp is preferably a color specified by law. The external notification device 50 may be controlled so that the light of the lamp is emitted in green when the mobile object 1 is moving in the predetermined region and the light of the lamp is emitted in blue when the mobile object 1 is moving on the roadway. When the external notification device 50 is a display device, the external notification device 50 displays text or a graphic indicating “moving on the sidewalk” when the mobile object 1 is moving in the predetermined region.
Returning to
The control device 100 includes, for example, a road type recognition unit 120, a physical object recognition unit 130, and a control unit 140. These constituent elements are implemented, for example, by a hardware processor such as a central processing unit (CPU) executing a program (software) 74. Also, some or all of these constituent elements may be implemented by hardware (including a circuit unit; circuitry) such as a large-scale integration (LSI) circuit, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU) or may be implemented by software and hardware in cooperation. The program may be pre-stored in the storage device 70 or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or a CD-ROM and installed in the storage device 70 when the storage medium is mounted in a drive device.
The road type recognition unit 120 recognizes whether the mobile object 1 is moving on the roadway or in the predetermined region. For example, the road type recognition unit 120 recognizes whether the mobile object 1 is moving on the roadway or in the predetermined region by analyzing an image captured by the external camera of the external environment detection device 10. Also, the output of a radar device, a LIDAR sensor, a sensor fusion device, or the like may be used in an auxiliary way. Details of a process of the road type recognition unit 120 will be described below.
The physical object recognition unit 130 recognizes a physical object located near the mobile object 1 on the basis of an output of the external environment detection device 10. Physical objects include some or all of mobile objects such as a vehicle, a bicycle, and a pedestrian, course boundaries such as a road marking, a step, a guardrail, a road shoulder, a median strip, structures installed on the road such as a road sign and a signboard, and an obstacle such as a falling object located (having fallen) on the course. For example, the physical object recognition unit 130 acquires information such as the presence, position, and type of another mobile object when an image captured by the external camera of the external environment detection device 10 is input to a trained model that has been trained to output information such as the presence, position, and type of a physical object if the image captured by the external camera is input. A type of the other mobile object can also be estimated on the basis of a size in the image and/or the intensity of reflected waves received by the radar device of the external environment detection device 10. Moreover, the physical object recognition unit 130 acquires, for example, the speed of another mobile object detected by the radar device using a Doppler shift or the like.
The control unit 140 controls, for example, the drive device 40 in accordance with a set driving mode. Also, the mobile object 1 may execute only some of the following driving modes, but the control unit 140 sets different speed limit values when the mobile object 1 moves on the roadway and when the mobile object 1 moves in the predetermined region in any case. In this case, the mode changeover switch 22 may be omitted.
In mode A-1, the control unit 140 controls the motor MT of the drive device 40 so that a distance from a physical object located in front of the mobile object 1 is maintained at a certain distance or more when the mobile object 1 moves on the roadway and the mobile object 1 moves at a first speed V1 (e.g., a speed of 10 [km/h] or more and less than several tens of kilometers per hour [km/h]) when a distance from a physical object located in front of the mobile object 1 is sufficiently long with reference to information about a course and a physical object based on the output of the physical object recognition unit 130. The control unit 140 controls the motor MT of the drive device 40 so that a distance from a physical object located in front of the mobile object 1 is maintained at a certain distance or more when the mobile object 1 moves in the predetermined region and the mobile object 1 moves at the second speed V2 (for example, a speed of less than 10 [km/h]) when a distance from a physical object located in front of the mobile object 1 is sufficiently long. This function is similar to an adaptive cruise control (ACC) function of a vehicle in which the first speed V1 or the second speed V2 is designated as the set speed and the technology used in ACC can be used. Moreover, in mode A-1, the control unit 140 controls the steering device SD so that a steering angle of the steered wheel is changed on the basis of a manipulation quantity of the manipulation element 14 such as the steering wheel. This function is similar to that of a power steering device and the technology used in the power steering device can be used. Also, electronic control is not performed in relation to steering and the mobile object 1 may include a steering device in which the manipulation element 14 and the steering mechanism are mechanically connected.
In mode A-2, the control unit 140 controls the steering device SD of the drive device 40 so that a target trajectory along which the mobile object 1 can move while avoiding a physical object within the course is generated with reference to information about the course and the physical object based on the output of the physical object recognition unit 130 and the mobile object 1 moves along the target trajectory. In relation to acceleration/deceleration, the control unit 140 controls the motor MT of the drive device 40 on the basis of the speed of the mobile object 1 and the manipulation quantity of the accelerator pedal or the brake pedal. The control unit 140 controls the motor MT of the drive device 40 in a state in which the first speed V1 is designated as the upper limit speed (wherein this control indicates that the mobile object 1 is not accelerated even if there is a further acceleration instruction when the mobile object 1 reaches the upper limit speed in the case of mode A-2) when the mobile object 1 is moving on the roadway and controls the drive device 40 in a state in which the second speed V2 is designated as the upper limit speed when the mobile object 1 is moving in the predetermined region.
In mode B, the control unit 140 controls the motor MT of the drive device 40 on the basis of the speed of the mobile object 1 and the manipulation quantity of the accelerator pedal or brake pedal. The control unit 140 controls the motor MT of the drive device 40 in a state in which the first speed V1 is designated as the upper limit speed (wherein this control indicates that the mobile object 1 is not accelerated even if there is a further acceleration instruction when the mobile object 1 reaches the upper limit speed in the case of mode B) when the mobile object 1 is moving on the roadway and controls the motor MT of the drive device 40 in a state in which the second speed V2 is designated as the upper limit speed when the mobile object 1 is moving in the predetermined region. In relation to steering, mode B is similar to mode A-1.
In mode C, the control unit 140 controls the drive device 40 so that a target trajectory along which the mobile object 1 can move while avoiding a physical object within the course is generated with reference to information about the course and the physical object based on the output of the physical object recognition unit 130 and the mobile object 1 moves along the target trajectory. Even in mode C, the control unit 140 controls the drive device 40 in a state in which the first speed V1 is designated as the upper limit speed when the mobile object 1 is moving on the roadway and controls the drive device 40 in a state in which the second speed V2 is designated as the upper limit speed when the mobile object 1 is moving in the predetermined region.
Hereinafter, a process of the road type recognition unit 120 will be described. The road type recognition unit 120 defines a plurality of virtual lanes including a virtual central lane including an assumed course of the mobile object 1, a virtual right lane located on the right side of the virtual central lane as seen from the mobile object 1, and a virtual left lane located on the left side of the virtual central lane as seen from the mobile object 1 in a space on a movement direction side of the mobile object 1 and recognizes whether the mobile object 1 is moving on a roadway or in a predetermined region on the basis of results of performing spatial classification processes on the basis of an output of the external environment detection device 10 for the plurality of virtual lanes.
The external environment detection device 10 used by the road type recognition unit 120 is an external camera configured to image the outside of the mobile object.
The road type recognition unit 120 sets the virtual central lane Lm, the virtual right lane Lr, and the virtual left lane Ll on the captured image IM, and performs a process for each region in the captured image IM. When this region is defined, if a series of edges indicating the course boundary (hereinafter referred to as course boundary edges) can be extracted by the road type recognition unit 120 in the captured image IM, the course boundary edges are designated as a boundary between two virtual lanes adjacent to each other among the plurality of virtual lanes. An edge is pixels (feature pixels) in which a difference between pixel values (luminance values, RGB values, and the like) from adjacent pixels is larger than a reference value. Although there are various methods of selecting adjacent pixels, when a line extending in a movement direction as seen from the mobile object 1 is extracted, it is preferable to select a pixel in a horizontal direction from a pixel of attention as an adjacent pixel.
When one of the course boundary edges that should be located on the left/right cannot be extracted, the road type recognition unit 120 designates a line offset by a specified width W (a width in an assumed plane seen from above) from the extracted course boundary edge to either the left or right as a boundary between two virtual lanes adjacent to each other.
When the edge line cannot be extracted at all, the road type recognition unit 120, for example, sets a line offset by a predetermined width (half of the specified width W) from the assumed course K to the left/right on the assumed plane, and designates a line on the plane of the captured image IM into which the set line is converted as a boundary between two virtual lanes adjacent to each other.
Thus, the road type recognition unit 120 sets a boundary between two virtual lanes adjacent to each other among the plurality of virtual lanes under the assumption that at least some of the plurality of virtual lanes are extended by a specified width on a road surface.
Moreover, the road type recognition unit 120 may perform an iterative process at a predetermined cycle and recognize whether the mobile object 1 is moving on the roadway or in the predetermined region by taking over previous recognition results for the plurality of virtual lanes.
The road type recognition unit 120 of the first embodiment recognizes whether the mobile object 1 is moving on the roadway or in the predetermined region on the basis of results of performing spatial classification processes on the basis of the output of the external environment detection device for the plurality of virtual lanes.
For example, the road type recognition unit 120 first recognizes whether the virtual central lane Lm is a representation of the roadway or the predetermined region. When a reliability degree of a recognition result related to the virtual central lane Lm is greater than or equal to a reference degree, the road type recognition unit 120 recognizes whether the mobile object 1 is moving on the roadway or in the predetermined region on the basis of the recognition result related to the virtual central lane Lm.
For example, the road type recognition unit 120 adds a point value to a roadway score Sr every time each of the plurality of first events indicating that the mobile object 1 is moving on the roadway in the region of the virtual central lane Lm of the captured image IM is recognized and recognizes that the mobile object 1 is moving on the roadway when the roadway score Sr is greater than or equal to a first threshold value Th1. At this time, the road type recognition unit 120 weights the point value in accordance with a confidence degree when each of the plurality of first events has been recognized and adds the weighted point value to the roadway score Sr. In this regard, when any one of a plurality of second events indicating that the mobile object 1 is moving on the sidewalk in the image captured by the external camera has been recognized, the road type recognition unit 120 recognizes that the mobile object 1 is moving on the sidewalk regardless of the roadway score Sr. The first events are an event in which there is no static obstacle such as a standing signboard, an event in which the vehicle is moving, an event in which there is a road surface marking, an event in which there is a crosswalk, and an event in which the vehicle is located on the lower side of the step. The second events are an event in which there is a static obstacle such as a standing signboard, an event in which there is a braille block, and an event in which the vehicle is located on the upper side of the step. In this case, a reliability degree of the recognition result related to the virtual central lane Lm is calculated on the basis of a value of the roadway score Sr. For example, the road type recognition unit 120 determines that the reliability degree of the recognition result related to the virtual central lane Lm is less than the reference degree when the roadway score Sr is less than the first threshold value Th1 and the second event is not recognized.
Alternatively, the road type recognition unit 120 may be configured to input the region of the virtual central lane Lm of the captured image IM to a first trained model, so that a result of recognizing whether the virtual central lane Lm, i.e., the region where the mobile object 1 is moving, is the roadway or the predetermined region is obtained. The first trained model is a model trained by machine learning so that information (an identification result) indicating whether a corresponding portion is a representation of the roadway or the predetermined region is output when an image having a size of the virtual central lane Lm is input. In this case, if the first trained model is set to output information indicating the reliability degree of the output information together with the identification result and the reliability degree output when the region of the virtual central lane Lm of the captured image IM is input to the first trained model is less than a reference value, it may be determined that the reliability degree of the recognition result related to the virtual central lane Lm is less than the reference degree.
When the reliability degree of the recognition result related to the virtual central lane Lm is less than the reference degree, the road type recognition unit 120 recognizes whether the mobile object 1 is moving on the roadway or in the predetermined region on the basis of results of performing spatial classification processes for the virtual left lane
Ll and the virtual right lane Lr. As the spatial classification process, the road type recognition unit 120 recognizes whether each of the virtual left lane Ll and the virtual right lane Lr is a representation of the roadway, a representation of the predetermined region, a representation of a region other than the course (e.g., a wall of a building), or unclear.
The road type recognition unit 120 recognizes whether the mobile object 1 is moving on the roadway or in the predetermined region on the basis of a combination of results of performing spatial classification processes for the virtual left lane Ll and the virtual right lane Lm. More specifically, the road type recognition unit 120 recognizes whether the mobile object 1 is moving on the roadway or in the predetermined region on the basis of a combination of identification results related to the virtual left lane Ll and the virtual right lane Lr and an analysis result related to the virtual central lane Lm.
As indicated by (3) to (5) of
As indicated by (3) of
As indicated by (4) of
As indicated by (5) of
As indicated by (6) to (8) of
First, as shown in (6) of
As indicated by (7) of
As indicated by (8) of
According to the first embodiment described above, it is possible to recognize whether the mobile object 1 is moving on the roadway or in the predetermined region different from the roadway by reflecting a feature of each virtual lane. Even in a case where it is difficult to make the determination only from the virtual central lane image, the recognition accuracy can be improved by referring to the virtual left lane image and the virtual right lane image. As a result, it is possible to appropriately recognize whether the mobile object 1 is moving on the roadway or in the predetermined region different from the roadway.
Hereinafter, a second embodiment will be described. In a control device 100 of the second embodiment, a road type recognition unit 120 uses an integrated trained model including a virtual lane setting layer to recognize whether a mobile object 1 is moving on a roadway or in a predetermined region different from the roadway.
When the captured image IM is input, the virtual lane setting layer outputs a range of a virtual central lane Lm, a virtual left lane Ll, and a virtual right lane Lr in the captured image IM. The virtual central lane recognition layer functions as in the first trained model in the first embodiment, the virtual left lane recognition layer functions as in the second trained model in the first embodiment, and the virtual right lane recognition layer functions as in the third trained model in the first embodiment. Although the recognition result integration layer has a function corresponding to the process shown in
By training the integrated trained model in this way, it is possible to recognize whether the mobile object 1 is moving on the roadway or in the predetermined region different from the roadway by reflecting a feature of each virtual lane. Even in cases where it is difficult to make the determination only from the virtual central lane image, the recognition accuracy can be improved by referring to the virtual left lane image and the virtual right lane image. Moreover, because recognition can be performed with features not assumed by a creator of a rule-based system using machine learning, there is a possibility that recognition accuracy can be further improved. As a result, according to the second embodiment, it is possible to appropriately recognize whether the mobile object 1 is moving on the roadway or in the predetermined region different from the roadway.
In the second embodiment, recognition results may be obtained by inputting the virtual central lane image, the virtual right lane image, and the virtual left lane image cut out from the captured image IM on the basis of the information of the virtual lane set on the basis of a rule as in the first embodiment to an integrated trained model # (in which the virtual lane setting layer is excluded from the layers shown in
The embodiment described above can be represented as follows.
A mobile object control device including:
Although modes for carrying out the present invention have been described above using embodiments, the present invention is not limited to the embodiments and various modifications and substitutions can also be made without departing from the scope and spirit of the present invention.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2022/016482 | 3/31/2022 | WO |
