Patent Application
20250214581
The present invention relates to a control device for a mobile object, a control method for a mobile object, and a storage medium.
In the related art, a motor-driven vehicle traveling support device that detects a direction in which a motor-driven vehicle can move from captured image data captured by an imaging unit and controls traveling of the motor-driven vehicle based on the detected direction has been disclosed (for example, see Patent Document 1).
However, the foregoing motor-driven vehicle traveling support device cannot recognize a type of road in a region of interest in some cases. In these cases, for example, traveling in accordance with types of roads cannot be achieved.
In view of such circumstances, an object of the present invention is to provide a control device for a mobile object, a control method for a mobile object, and a storage medium capable of recognizing a type of road in a region of interest with high accuracy and realizing control of a mobile object in accordance with the type of road.
A control device for a mobile object, a control method for a mobile object, and a storage medium according to aspects of the present invention have the following configurations.
According to the aspects (1) to (16), the control device can realize the recognizing of a type of road in the region of interest with high accuracy and the controlling of the mobile object in accordance with the type of road.
According to the aspect (12), the control device can realize the recognizing of a type of road with high accuracy and the controlling of the mobile object in accordance with the type of road while reducing a processing load by changing a process of recognizing the type of road in accordance with a result of recognition of a direction of the bicycle mark.
Hereinafter, embodiments of a control device for a mobile object, a control method for a mobile object, and a storage medium according to the present invention will be described with reference to the drawings.
Hereinafter, the embodiments of the control device for the mobile object, the control method for the mobile object, and the storage medium according to the present invention will be described with reference to the drawings. A mobile object moves on both a roadway and a predetermined region different from the roadway. The mobile object is also referred to as a micro mobility. An electric kick scooter is a type of micromobility. The predetermined region is, for example, a sidewalk. The mobile object may be a vehicle which a passenger can board or an autonomous mobile object that can perform unmanned autonomous traveling. The latter autonomous mobile object is used for convey luggage or the like. The mobile object may include a small mobility that carries luggage or leads people while walking alongside a user as well as other autonomous mobile objects (such as walking robots). The predetermined region may include some or all of roadside strips, bicycle lanes, and open spaces or may include some or all of sidewalks, roadside strips, bicycle lanes, and open spaces. In the following description, the predetermined region is assumed to be a sidewalk. In the following description, a portion described as “sidewalk” can be read as a “predetermined region” as appropriate.
The external detection device 10 is any of various devices in which a traveling direction of the mobile object 1 is a detection range. The external detection device 10 includes an external camera, a radar device, a light detection and ranging (LIDAR), and sensor fusion device. The external detection device 10 outputs information (an image, a position of an object, or the like) indicating a detection result to the control device 100.
The mobile object sensor 12 includes, for example, a speed sensor, an accelerations sensor, a yaw rate (angular velocity) sensor, an azimuth sensor, and an operation amount detection sensor mounted on the operator 14. The operator 14 includes, for example, an operator (for example, an acceleration pedal or a brake pedal) that gives an instruction for acceleration or deceleration and an operator (for example, a steering wheel) that instructs steering. In this case, the mobile object sensor 12 may include an accelerator opening sensor, a brake depression amount sensor, and a steering torque sensor. The mobile object 1 may include another type of operator (for example, a rotational operator (that is not annular), a joystick, a button, or the like) as the operator 14.
The internal camera 16 images at least a head of a passenger of the mobile object 1 from the front. The internal camera 16 is a digital camera in which an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) is used. The internal camera 16 outputs a captured image to the control device 100.
The positioning device 18 is a device that positions the mobile object 1. The positioning device 18 is, for example, a global navigation satellite system (GNSS) receiver, and identifies a position of the mobile object 1 based on a signal received from a GNSS satellite and outputs the position as positional information. The positional information of the mobile object 1 may be estimated from a position of a Wi-Fi base station to which a communication device to be described below is connected.
The mode changeover switch 22 is a switch that is operated by a passenger. The mode changeover switch 22 may be a mechanical switch or a graphical user interface (GUI) switch set on a touch panel. The mode changeover switch 22 receives an operation of switching a driving mode to one of mode A that is an assist mode in which one of a steering operation and an acceleration or deceleration control is performed by a passenger and the other thereof is automatically performed, and may be mode A-1 in which a steering operation is performed by a passenger and acceleration or deceleration speed control is automatically performed or may be a mode A-2 in which an acceleration or deceleration speed operation is performed by a passenger and steering control is automatically performed; mode B that is a manual driving mode in which a steering operation and an acceleration or deceleration speed operation are performed by a passenger; and mode C that is an automated driving mode in which steering control and acceleration or deceleration speed control are automatically performed.
The mobile mechanism 30 is a mechanism that moves the mobile object 1 on a road. The mobile mechanism 30 is, for example, a wheel group including a steering wheel and a driving wheel. The mobile mechanism 30 may also be legs for multi-legged walking.
The driving device 40 outputs a force to the mobile mechanism 30 to move the mobile object 1. For example, the driving device 40 includes a motor that drives a driving wheel, a battery that stores power to be supplied to the motor, and a steering device that adjusts a steering angle of a steering wheel. The driving device 40 may include an internal combustion engine or a fuel cell as driving force output means or power generation means. The driving device 40 may further include a brake device using a frictional force or air resistance.
The external report device 50 is provided, for example, on an outer plate of the mobile object 1 and is a lamp, a display device, a speaker, or the like that reports information to the outside of the mobile object 1. The external report device 50 performs different operations according to whether the mobile object 1 is moving on a sidewalk or the mobile object 1 is moving on a roadway. For example, the external report device 50 is controlled such that a lamp is caused to emit light when the mobile object 1 is moving on a sidewalk and the lamp is caused not to emit light when the mobile object 1 is moving on a roadway. A light color of the lamp is preferably a color defined by law. The external report device 50 may be controlled such that the lamp is caused to emit green light when the mobile object 1 is moving on a sidewalk and the lamp is caused to emit blue light when the mobile object 1 is moving on a roadway. When the external report device 50 is a display device, the external report device 50 displays text or graphics indicating “moving on a sidewalk” when the mobile object 1 is traveling on the sidewalk.
Referring back to
The control device 100 includes, for example, a road type recognition unit 120, a recognition unit 130, and a control unit 140. These constituent elements are realized, for example, by causing a hardware processor such as a central processing unit (CPU) to execute a program (software) 74. Some or all of the constituent elements may be realized by hardware (including a circuitry) such as a large scale integration (LSI), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU) or may be realized in cooperation between software and hardware. The program may be stored in advance in the storage device 70, or may be stored in a detachable storage medium (non-transitory storage medium) such as a DVD or a CD-ROM and the storage medium may be mounted on a drive device to be installed in the storage device 70.
The road type recognition unit 120 recognizes whether the mobile object 1 is moving on a roadway or a sidewalk. The road type recognition unit 120 recognizes whether the mobile object 1 is moving on a roadway or a sidewalk, for example, by analyzing an image captured by an external camera of the external detection device 10. The external camera may be a plurality of cameras (for example, stereo cameras). An output of a radar device, an LIDAR, a sensor fusion device, or the like may be used supplementarily.
Based on an image captured by the external camera, the road type recognition unit 120 recognizes whether the mobile object 1 is moving on a roadway or a sidewalk. For example, the road type recognition unit 120 classifies pixels in a frame of an image by semantic segmentation into classes, assigns labels, and assumes a plurality of virtually divided regions and boundary lines therebetween. The road type recognition unit 120 classifies at least regions of an image captured by the external camera into a region where there is the mobile object 1 (hereinafter referred to as a self-region) and a region which is adjacent to the self-region and where there is no mobile object 1 (hereinafter referred to as an adjacent region) to perform a process. For example, the road type recognition unit 120 recognizes whether a region of interest (for example, a self-region) is a sidewalk or a roadway by using a bicycle mark displayed on a road. The details of the process will be described below.
The recognition unit 130 recognizes objects that are around the mobile object 1 based on an output of the external detection device 10. The objects include some or all of mobile objects such as pedestrians, vehicles and bicycles, runway boundaries such as lane marking lines, steps, guardrails, road shoulders and median strips, structures provided on roads, such as road marks (signs) or signboards, and obstacles such as debris located (fallen) on runways. For example, the recognition unit 130 acquires information such as presence, positions, types, and the like of other mobile objects by inputting the images captured by the external camera to a trained model that is trained to output information such as presence, positions, types, and the like of objects when images captured by the external camera of the external detection device 10 are input. The types of other mobile objects can also be estimated based on sizes of the objects in the images, intensity of reflected waves received by a radar device of the external detection device 10, and the like. For example, the recognition unit 130 acquires speeds of other mobile objects detected by the radar device using Doppler shift or the like.
For example, the control unit 140 controls the driving device 40 in accordance with a set driving mode. The mobile object 1 may perform only some of the following driving modes and the control unit 140 causes speed limit values to be different between cases in which the mobile object 1 is moving on a roadway and is moving on a sidewalk in any case. In this case, the mode changeover switch 22 may be omitted.
In mode A-1, the control unit 140 controls the motor MT of the driving device 40 such that a distance to an object located in front of the mobile object 1 is kept at a given distance or more when the mobile object 1 is moving on a roadway with reference to information regarding a runway and the object based on an output of the recognition unit 130, and the mobile object 1 is moving at a first speed V1 (for example, a speed equal to or greater than 10 [km/h] and less than tens of [km/h]) when the distance to the object located in front of the mobile object 1 is sufficiently long.
The control unit 140 controls the motor MT of the driving device 40 such that the distance to the object located in front of the mobile object 1 is kept at the given distance or more when the mobile object 1 is moving on a roadway and the mobile object 1 is moving at a second speed V2 (for example, a speed less than 10 [km/h]) when the distance to the object located in front of the mobile object 1 is sufficiently long. This function is similar to an adaptive cruise control (ACC) function of a vehicle that has the first speed V1 or the second speed V2 as a set speed, and a technology used in ACC can be used. In mode A-1, the control unit 140 controls the steering device SD such that a steering angle of the steering wheel is changed based on an operation amount of the operator 14 such as a steering wheel. This function is similar to a function of a power steering device and a technology used in a power steering device can be used. For steering, the mobile object 1 may include a steering device to which the operator 14 and the steering mechanism are mechanically connected without performing electronic control.
In mode A-2, the control unit 140 generates a movable target trajectory in which movement is possible while avoiding an object in a runway with reference to information regarding the runway and the object based on an output of the recognition unit 130 and controls the steering device SD of the driving device 40 such that the mobile object 1 moves along the target trajectory. For an acceleration or deceleration speed, the control unit 140 controls the motor MT of the driving device 40 based on a speed of the mobile object 1 and an operation amount of the acceleration pedal or the brake pedal. The control unit 140 controls the motor MT of the driving device 40 using the first speed V1 as an upper limit speed when the mobile object 1 is moving on a roadway (which, in the case of mode A-2, means that the mobile object 1 is not accelerated even when an additional acceleration instruction is given when a speed reaches the upper limit speed) and controls the driving device 40 using the second speed V2 as an upper limit speed when the mobile object 1 is moving on a sidewalk.
In mode B, the control unit 140 controls the motor MT of the driving device 40 based on a speed of the mobile object 1 and an operation amount of the acceleration pedal or the brake pedal. The control unit 140 controls the motor MT of the driving device 40 using the first speed V1 as the upper limit speed when the mobile object 1 is moving on a roadway (which, in the case of mode B, means that the mobile object 1 is not accelerated even when an additional acceleration instruction is given when a speed reaches the upper limit speed) and controls the driving device 40 using the second speed V2 as an upper limit speed when the mobile object 1 is moving on a sidewalk. The steering is similar to that of mode A-1. Hereinafter, a mode in the case of traveling on a roadway is referred to as mode B1 (second movement mode) and a mode in the case of traveling on a sidewalk is referred to as a mode B2 (first movement mode).
In mode C, the control unit 140 generates a movable target trajectory in which movement is possible while avoiding an object in a runway with reference to information regarding the runway and the object based on an output of the recognition unit 130 and controls the driving device 40 such that the mobile object 1 moves along the target trajectory. In mode C, the control unit 140 controls the driving device 40 using the first speed V1 as the upper limit speed when the mobile object 1 is moving on a roadside, and controls the driving device 40 using the second speed V2 as the upper limit speed when the mobile object 1 is moving on a sidewalk
The road type recognition unit 120 recognizes whether a region of interest is a roadway or a sidewalk based on a bicycle mark for guiding passage of a bicycle detected by the external detection device 10. The control unit 140 controls the mobile object 1 based on a recognition result recognized by the road type recognition unit 120. For example, the control unit 140 controls the mobile object 1 in a roadway mode for traveling on a roadway when the mobile object 1 is located on a road recognized as a roadway by the road type recognition unit 120. The roadway mode is a mode in which the upper limit speed during traveling is limited to the first speed. The control unit 140 controls the mobile object 1 in a sidewalk for traveling on a sidewalk when the mobile object 1 is located on a road recognized as a sidewalk by the road type recognition unit 120. The sidewalk mode is a mode in which the upper limit speed during traveling is limited to the second speed.
The road type recognition unit 120 recognizes whether the region of interest is a roadway or a sidewalk based on a bicycle mark and a direction of the bicycle mark.
The storage device 70 stores feature information of various bicycle marks such as the bicycle marks B1 and B2. The feature information is information (information indicating a distribution of edges, a distribution of luminance values, or the like) indicating features obtained by performing an image recognition process on captured images of the bicycle marks B1 and B2. The road type recognition unit 120 recognizes a bicycle mark or a direction indicated by the bicycle mark based on the feature information obtained from the captured image of the bicycle mark B1 or B2 and the feature information stored in the storage device 70.
In the foregoing process, widths of roads may be added. The road type recognition unit 120 may recognize whether a region of interest may be a roadway or a sidewalk based on a bicycle mark and widths of roads on the right and left sides of the bicycle mark. The road type recognition unit 120 recognizes a road with a narrow width between a first road including a bicycle mark and a second road adjacent to the first road as a sidewalk and recognizes a road with a broad width as a roadway.
The second road adjacent to the first road includes, for example, the second road adjacent to the first road via a boundary object for distinguishing the first road from the second road. The boundary object is, for example, an object with a narrow width provided at a boundary between a sidewalk and a roadway, such as a curbstone, a fence, a guardrail, a planting, or a marking line displayed on a road.
For example, as in
When a bicycle mark is provided on the first road 1 (the road R1 or R2) and a boundary object for distinguishing the first road from the second road adjacent to the first road is adjacent to the bicycle mark, the road type recognition unit 120 may recognize whether the second road is a roadway or a sidewalk based on one or both of presence or absence of vehicles on the first and second roads and the width of the first road and the width of the second road. This is because the widths are required with higher accuracy when there is the boundary object. As will be described below, a road on which there is a vehicle is determined to be a roadway and a road on which there is no vehicle is determined to be a sidewalk.
As illustrated in
The road type recognition unit 120 may recognize whether a region of interest is a roadway or a sidewalk based on a sign provided in the vicinity of a bicycle mark.
The bicycle mark B3 does not have a mark indicating a traveling direction. For example, when the bicycle mark B3 is displayed on a sidewalk, a bicycle can travel in either direction. For example, a bicycle can travel in the direction D1 and can also travel in an opposite direction to the direction D1. Therefore, even when the road type recognition unit 120 recognizes the bicycle mark B3, it may be difficult to recognize the region of interest is a roadway or a sidewalk since there is no direction of the bicycle mark, as described above.
Thus, based on a sign C provided in the vicinity of the bicycle mark, the road type recognition unit 120 recognizes whether he region of interest is a roadway or a sidewalk. The vicinity of the bicycle mark is a sign provided in a guide lane or in the vicinity of the guide lane.
The road type recognition unit 120 recognizes whether a region of interest is a roadway or a sidewalk based on the bicycle mark B3 and the sign described above. In the foregoing example, when the bicycle mark does not include information indicating a guide direction, the road type recognition unit 120 refers to the sign, as described above. Instead of this, even when the bicycle mark includes information indicating the guide direction, a type of road in the region of interest may be recognized with reference to the sign.
The road type recognition unit 120 can recognize a type of road with higher accuracy by using the bicycle mark and the sign described above.
The road type recognition unit 120 may recognize whether a region of interest is a roadway or a sidewalk by using a bicycle mark and various elements included in a road.
For example, the road type recognition unit 120 determines that a region satisfying many of the following conditions is a roadway and a region not satisfying many of the following conditions is a sidewalk. The conditions are, for example, conditions that there is no static obstacle, the vehicle M stops or is moving, there is no pedestrian, there is the road surface mark RM on a road surface, and there is a pedestrian crossing PC. It may be determined that the boundary object OB is a curbstone and a region in a direction in which a step difference of the curbstone descends is a roadway.
For example, when the boundary object OB descends from the road R2 to the road R1 as in AT indicating a cross-sectional view of a curbstone seen in an X direction of
The road type recognition unit 120 may recognize that the region of interest is a roadway when there is the bicycle mark and a predetermined number of conditions (one or more conditions) is satisfied among the foregoing conditions (see
The road type recognition unit 120 may derive a score using information (the bicycle mark and various elements) obtained from a result of the recognition of the recognition unit 130 and may recognize whether the region of interest is a roadway or a sidewalk based on the derived score, as illustrated in
The road type recognition unit 120 compares a total score obtained by totalizing the scores derived in this way with thresholds of the scores and recognizes whether a region of interest is a roadway or a sidewalk. For example, when the total score is a score indicating that the region of interest is similar to a roadway and the total score is equal to or greater than a threshold, the road type recognition unit 120 recognizes that the region of interest is the roadway. For example, when many above-described conditions are satisfied, a higher total score tends to be derived than when many above-described conditions are not satisfied.
For example, when the total score is a score indicating that the region of interest is similar to a sidewalk and the total score is equal to or greater than a threshold, the road type recognition unit 120 recognizes that the region of interest is the sidewalk. For example, when many above-described conditions are not satisfied, a higher total score tends to be derived than when many above-described conditions are satisfied.
Subsequently, if the road type recognition unit 120 recognizes that the region of interest is the roadway, the control unit 140 causes the mobile object 1 to move in the roadway mode when the mobile object 1 moves in the region of interest (steps S108 and S110). If the road type recognition unit 120 recognizes that the region of interest is the sidewalk, the control unit 140 causes the mobile object 1 to move in the sidewalk mode when the mobile object 1 moves in the region of interest (steps S108 and S112). In this way, the process of one routine of the flowchart ends.
As described above, the control device 100 can recognize a type of road in the region of interest with high accuracy. Accordingly, the control device 100 can control the mobile object 1 according to an aspect appropriate for a surrounding situation.
If the direction of the bicycle mark can be recognized, the road type recognition unit may recognize whether the region of interest is a roadway or a sidewalk based on the bicycle mark and the direction of the bicycle mode. If the direction of the bicycle mark cannot be recognized, the road type recognition unit may recognize whether the region of interest is a roadway or a sidewalk based on one or both of the bicycle mark and the widths of the right and left roads of the bicycle mark and the sign provided in the vicinity of the bicycle mark.
If the bicycle mark and the direction of the bicycle mark cannot be recognized, the road type recognition unit 120 recognizes a type of road based on the bicycle mark and various elements of the road (step S206).
Subsequently, if the road type recognition unit 120 recognizes that the region of interest is the roadway, thee control unit 140 causes the mobile object 1 to move in the roadway mode when the mobile object 1 moves in the region of interest (steps S208 and S210). If the road type recognition unit 120 recognizes that the region of interest is the sidewalk, the control unit 140 causes the mobile object 1 to move in the sidewalk mode when the mobile object 1 moves in the region of interest (steps S208 and S212). Accordingly, the process of one routine of the flowchart ends.
According to the foregoing modified example, the control device 100 can recognize the type of road with high accuracy and can realize control of the mobile object in accordance with the type of road while reducing a processing load by changing the process of recognizing the type of road in accordance with the result of the recognition of the direction of the bicycle mark.
In the foregoing example, the road type recognition unit 120 derives the score in accordance with the result of the recognition by applying the method of deriving the score in accordance with the result of the recognition, as described above. However, instead of this, the total score may be derived using a trained model. The trained model is a model that outputs a score in accordance with input information when the bicycle mark, the direction of the bicycle mark, and information regarding various elements are input. The trained model is a model in which learning data is learned. The learning data is information in which information such as the bicycle mark, the direction of the bicycle mark, information regarding various elements is associated with a type of road corresponding to information (answer data indicating a roadway or a sidewalk). The trained model outputs a high score, for example, when a score for estimating that the region of interest is a roadway is output and the information such as the bicycle mark corresponding to the roadway, the direction of the bicycle mark, and the information regarding various elements is input.
The trained model may be a model that outputs a score indicating whether a region indicated by an image is a roadway or a sidewalk when the image is input. The trained model outputs a score indicating that a region of interest is a roadway, for example, when an image of the roadway is input, and outputs a score indicating that the region of interest is a sidewalk, for example, when an image of the sidewalk is input. Learning data used to train the trained model is, for example, information in which an image of a roadway including information indicating the bicycle mark and information indicating the direction of the bicycle mark (an image of a roadway) and a type of road corresponding to the information (for example, answer data indicating the roadway) are associated. A score in which information indicating the bicycle mark and the direction of the bicycle mark are considered is output by learning the learning data.
In the embodiment, the bicycle mark may be used to determine a type of road. For example, when an image is input, the trained model derives a score for estimating a roadway or a sidewalk. The road type recognition unit 120 may correct the derived score using presence or absence of the bicycle mark or the direction of the bicycle mark and recognize a type of road based on the corrected result. For example, even when a score for estimating a region of interest is a sidewalk is derived and a bicycle mark indicating a direction of a bicycle (for example, the bicycle mark B1 in
The control device 100 may apply a process of recognizing whether a region of interest is a roadway or a sidewalk using a bicycle mark in various modes (modes A to C). For example, in the case of movement on a sidewalk in various modes, a limit speed may be limited to the second speed.
As described above, the control device 100 can realize control of a mobile object in accordance with a type of road by recognizing the type of road in the region of interest with high accuracy based on the bicycle mark for guiding passage of a bicycle detected by a detection device that detects an external situation of the mobile object.
The above-described embodiment can be expressed as follows:
The embodiments for carrying out the present invention have been described above, but the present invention is not limited to the embodiments. Various modifications and substitutions can be made within the scope of the present invention without departing from the gist of the present invention.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2022/015397 | 3/29/2022 | WO |
