Patent Application
20220182498
The present invention relates to a system making a decision based on data communication.
Increase in level of automation increases demand for computational capability on an edge side. Computational and decision-making capabilities of autonomous systems face the challenge of dealing with unknown obstacle situations. It is desirable to assist and support secure and optimal decision making of the autonomous systems and reduce a burden on the computational capability.
PTL 1 describes an example of a system that makes a decision based on data communication. The system identifies an area on a map, corresponding to a portion within a distance threshold value.
The system compresses images in different areas with different data compression ratios.
Unfortunately, the system requires a high computational capability due to use of feature matching between a map and an image, which conflicts with real-time performance requirements.
The system also transmits a compressed image to a remote system. This can be achieved when traffic is not congested or when an effective communication rate is high. However, when traffic is congested, an excessive load is applied on a communication network to limit the amount of data that can be transmitted, and thus the system may not operate efficiently.
One of limitations of PTL 1 is that there is no description of a method for reducing data to reduce the network load.
Second, PTL 1 does not describe a decision-making technique. For example, there is no description of how the system determines which data are to be transmitted based on a vehicle state, a driving scenario, a vehicle purpose, a network availability, and so on.
Finally, PTL 1 describes a difficult scenario in which a vehicle can benefit from decision-making capability of a human operator or a computing system with higher performance.
To allow a system, which is fully autonomous, partially autonomous, or semi-autonomous, to operate safely, continuous communication or connection with a remote system, such as a supervisory system, is required.
PTL 1: US 2016/0283804 A
In particular, conventional techniques each have a problem that the amount of data to be communicated is large.
The present invention is made to solve such a problem, and an object of the present invention is to provide a system making a decision based on data communication and being capable of reducing the amount of data to be communicated.
A system according to the present invention makes a decision based on data communication, and includes a function of acquiring a map image, a function of determining a first area and a second area in the map image, a first transmission determination function of determining whether to transmit data related to the first area through a communication network, a second transmission determination function of determining whether to transmit data related to the second area through the communication network, a function of detecting objects around the system, a function of determining a position in the map image for each of the objects detected, a function of determining whether each of the objects detected belongs to the first area, based on the position of the corresponding one of the objects in the map image, a function of determining whether each of the objects detected belongs to the second area, based on the position of the corresponding one of the objects in the map image, a compression ratio determination function of determining a data compression ratio for each of the objects detected, based on a distance to the corresponding one of the objects, a function of compressing data related to each of the objects detected in accordance with the data compression ratio of the corresponding one of the objects to generate compression data related to the corresponding one of the objects, a function of transmitting the compression data related to each of the objects belonging to the first area through the communication network when data related to the first area is determined to be transmitted, a function of transmitting the compression data related to each of the objects belonging to the second area through the communication network when data related to the second area is determined to be transmitted, a function of receiving reply data replied in association with the compression data transmitted, through the communication network, and a function of making a decision in accordance with the reply data.
The system according to the present invention makes a decision based on data communication, and includes a processor that is capable of: acquiring a map image; determining a first area and a second area in the map image; determining whether to transmit data related to the first area through a communication network as a first transmission determination; determining whether to transmit data related to the second area through the communication network as a second transmission determination; detecting objects around the system, a function of determining a position in the map image for each of the objects detected; determining whether each of the objects detected belongs to the first area, based on the position of the corresponding one of the objects in the map image, a function of determining whether each of the objects detected belongs to the second area, based on the position of the corresponding one of the objects in the map image; determining a data compression ratio for each of the objects detected, based on a distance to the corresponding one of the objects; compressing data related to each of the objects detected in accordance with the data compression ratio of the corresponding one of the objects to generate compression data related to the corresponding one of the objects; transmitting the compression data related to each of the objects belonging to the first area through the communication network when data related to the first area is determined to be transmitted; transmitting the compression data related to each of the objects belonging to the second area through the communication network when data related to the second area is determined to be transmitted; receiving reply data replied in association with the compression data transmitted, through the communication network; and making a decision in accordance with the reply data.
The present specification includes the disclosure of Japanese Patent Application No. 2019-051272, which is the basis of the priority of the present application.
The system according to the present invention appropriately determines not only whether to transmit data on objects but also a data compression ratio of each of the objects, so that the amount of data to be communicated can be reduced.
Specific examples of the present invention can individually obtain effects below as examples.
An onboard computing platform (edge side computing platform) can sample, filter, and compress sensor data before transmitting it to a remote system. The edge-side computing platform can also receive an operation instruction from a remote system for secure and optimal decision making. The remote system may be, for example, a remote assistance system, which may involve a trained human operator, or may be a computing platform with high computational capability. The remote assistance system can provide a secure and optimal operation instruction to the edge side system requesting assistance.
The edge side system can receive the secure and optimal operation instruction from the remote system in real time without delay. This is especially effective in the following situations where,
a vehicle itself cannot make a secure and optimal decision,
the vehicle wants to pass control to a secure driver, but the secure driver is unaware,
the vehicle has encountered an unknown or unexplained failure situation,
the vehicle has a failure in a function, an operation, or a system,
sensor data in the vehicle needs to be uploaded for learning to improve the decision-making capability of a remote system, and
an occupant or passenger in the vehicle requests assistance.
In any of the above situations, the remote system may require enormous information on vehicle conditions and driving scenarios to make secure and optimal decisions. Thus, a principle is to use a map of the surrounding environment and update static and dynamic information on the map to make secure and optimal decisions. In an embodiment of the present invention, the edge-side system classifies the vehicle environment into a high-risk area (a travelable area) and a low-risk area (a static map area, a portion including a landmark on the map, a building that is not a part of a road network/graph, etc.) based on the map and the positional information on the vehicle. Then, the edge-side system can determine whether to update or transmit a dynamic traffic participant in the vehicle environment to the remote assistance system based on accuracy of a position of the vehicle, accuracy of conditions (vehicle position, speed, throttle, braking, steering) of the vehicle, and the map. Next, the edge-side system performs a clustering operation based on information on a detected object in the filtered vehicle environment, and then identifies a convex hull surrounding each cluster. Then, the edge-side system performs a cropping process of the detected object cluster in each area from data on the vehicle environment. The edge-side system finally selects an adaptive compression ratio for the object cluster detected and cropped based on an effective communication rate of a network, a distance from an environmental recognition sensor module to the object cluster detected, and a driving scenario.
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The present invention can be implemented as a system for making a decision based on data communication. Systems, functions, and methods, described herein are exemplary and do not limit the scope of the invention. Each aspect of the systems and methods disclosed herein can be configured in a variety of different combinations of configurations, all of which are assumed herein.
In each embodiment, a particular component or description can be replaced with a component or description in another embodiment. For example, those skilled in the art can achieve details of a certain process in a first embodiment according to a specific example described in a second embodiment.
A configuration according to the first embodiment provides a method for improving or assisting completely autonomous or semi-autonomous operation of a vehicle by receiving an operation instruction or assistance from a remote assistance system. The remote assistance system may include a human operator or a computing platform with high computational capability. The vehicle may provide sensor data to the remote assistance system to receive an operation instruction or assistance from the remote assistance system. The sensor data includes an image or a video stream of vehicle environment, light detection and ranging, or laser imaging detection and ranging (LIDAR) data, radio detection and ranging (RADAR) data, and the like. In contrast, the remote assistance system may assist the vehicle in detecting, classifying, or predicting behavior of an object, and assist in making a secure and optimal decision in any driving scenario. Thus, the vehicle can benefit from secure and optimal decision-making capability of a remote human operator, or high computational capability of a remote assisted computing platform.
Examples of a rare driving scenario in which a vehicle may require decision-making capability of a remote human operator or high computational capability of a remote assisted computing platform include the following case. In the case, the vehicle requires a vehicle position determining unit to execute a function requiring high computational capability that does not converge within a required limit and cannot be executed using an onboard computing platform. In such a situation, the vehicle may require assistance from a remote assistance system with high computational capability to perform the function. This causes the vehicle to upload sensor data to the remote assistance system with high computational capability, thereby receiving highly accurate positional information.
In another example, an onboard decision-making unit of a vehicle may require an onboard secure driver to take over control of the vehicle. However, the secure driver is unaware of this or is not careful about this, and thus may not receive control within a predetermined time frame, which can lead to an accident. In such a scenario, the vehicle can require remote assistance to take over vehicle control because the secure driver is not careful.
In another example, onboard detection of a vehicle or a decision-making and planning unit encounters an unknown situation or an unknown obstacle, and is not confident enough for the vehicle to make a secure operation decision. In such a case, the vehicle may request remote assistance. Similarly, when the onboard detection and a recognition system fail to detect a potential obstacle in real time, or when the vehicle encounters an unknown obstacle, this situation may lead to a traffic accident, and then a user and a passerby may be injured. Thus, the vehicle can upload sensor data on the situation to the remote assistance system and receive a secure and optimal operation instruction for the sensor data.
In yet another example, the vehicle may need to upload its sensor data to a cloud for online learning. This is to improve decision-making capability, detection, etc. In such a scenario, a bandwidth restriction or another data communication restriction may prohibit real-time uploading of sensor data. Such a scenario may cause compression of sensor data to degrade performance. Thus, in such a scenario, applying an embodiment of the present invention enables vehicle sensor data to be uploaded in real time without losing detailed information.
When the remote assistance system assists the vehicle, the remote assistance system may request various data representing an environment around the vehicle in real time to make a secure and optimal decision. For example, when a remote human operator takes over control of the vehicle remotely, video or image data representation of the surroundings of the vehicle is required to make a secure decision. However, a platform with high computational capability may require sensor data to make a secure and optimal decision.
In view of the above example, there are provided a method and a function for sampling, filtering, and compressing sensor data representing the vehicle environment before it is transmitted and uploaded to the remote assistance system. In one example, the vehicle receives an image of the environment from a camera mounted on the vehicle. The vehicle may receive a map of the environment (lane information, a stop line, etc.) such as a vector map. The map may include strength of an environment during navigation and an image file. The map may also include various road structural features and locations. The vehicle may receive a global position and its state (global speed, direction, acceleration, etc.). The vehicle may also identify itself or determine its position on the map based on its condition and position. The vehicle may divide the map into high-risk and low-risk areas based on a position of the vehicle on the map. In one example, the high-risk area may include an area related to driving conditions of the vehicle (a road on which the vehicle is traveling and the vicinity of the road). Then, the vehicle may determine importance and priority of updating the remote assistance system with high-risk area information, low-risk area information, or both, based on a position of the vehicle and accuracy of conditions thereof. For example, when a position of the vehicle is within an acceptable threshold value, the vehicle may determine to transmit only an object cluster detected and cropped in the high-risk area. One of reasons behind such decision-making is that the low-risk area contains a structural or landmark feature or a static feature that is useful for determining a position of the vehicle. In contrast, the high-risk area is important in decision-making in driving. The vehicle may also identify an object in an environment with the help of an object detection sensor and its function. After the object is identified, the vehicle may perform a clustering function for clustering the detected object based on a Euclidean distance, a class, or an object feature. After clustering the detected object, the vehicle may determine a boundary box or convex hull that surrounds each cluster. Then, the vehicle may crop each of object clusters detected in the high-risk and low-risk areas from the sensor data. Finally, the vehicle may determine a different compression ratio for each cluster based on a driving scenario and a bandwidth restriction of the vehicle. When a bandwidth availability is very low, the vehicle may transmit only boundary box or convex hull information for each detected object cluster.
In some cases, the functions described herein may be based on sensor data other than camera sensor data. For example, the sensor data may come from various sensors such as a LIDAR sensor, a RADAR sensor, an ultrasonic sensor, and an audio sensor. When a computing platform mounted on the vehicle allows fusion of multiple sensors, fusion sensor data may be used. In the case of object detection and a convex hull estimation unit, any available configuration can be used. In one example, the LIDAR sensor provides point cloud data for the environment, and the point cloud data represents an object in the environment. LIDAR information can be used for clustering and convex hull estimation. After that, a detected object cluster may be cropped from LIDAR data, and then the decision-making unit may determine the importance and priority of the detected and cropped object cluster. After the importance is determined, a bandwidth-based compression unit may determine the compression ratio of each of detected and cropped object cluster before the importance is transmitted to the remote assistance system. A similar method can be used for RADAR sensor data, and the same applies to multiple sensor fusion data.
Hereinafter, an example of the system according to the first embodiment will be described in detail. An example of a system for making a decision based on data communication will be described using an automobile. However, the present invention can also be implemented in other systems, and can also be applied to, for example, vehicles (passenger cars, buses, trucks, trains, golf carts, etc.), industrial machines (construction machines, farm machines, etc.), robots (ground robots, water robots, warehouse robots, service robots, etc.), aircraft (fixed-wing aircraft, rotary-wing aircraft, etc.), and ships (boats, ships, etc.). The present invention can also be applied to vehicles other than these.
Compression and transmission of the image 500 may not work well because of a bandwidth restriction. A high compression ratio leads to information loss. Maps used for driving continue to increase in amount of information. To make a secure and optimal decision, it may be sufficient to upload only dynamic information in the vehicle environment for remote assistance. Thus, the vehicle environment captured by the sensors mounted on the vehicle is sampled, filtered, compressed, and transmitted. In the case of the image 500, the traffic participants 506, 508, 509, 511, 512, 520, and 521 (
A second embodiment is achieved by adding a more specific description and adding or changing some configurations and operations in the first embodiment.
The calculation means 701 includes, for example, a processor. The storage means 702 includes a storage medium such as a semiconductor memory or a magnetic disk device. The communication means 703 includes input-output means such as an input-output port or a communication antenna. The communication means 703 can perform wireless communication through, for example, a wireless communication network. The system 700 can communicate with an external computer (e.g., a remote assistance system or a decision-making system mounted on another vehicle) using the communication means 703. The system 700 may include input-output means other than the communication means 703.
The system 700 has functions of performing the respective processes illustrated in
The system 700 can be mounted on, for example, a vehicle (the vehicle 200 illustrated in
The system 700 may be mounted in a configuration other than a vehicle. The system 700 may be mounted on a vehicle other than that illustrated in
Hereinafter, the vehicle 200 illustrated in
The sensors include a distance sensor that measures a distance to an object around the vehicle 200. The distance sensor may include a RADAR sensor. The example of
The sensors may also include an image sensor (imaging means) that captures an image of surroundings of the vehicle 200. The example of
The sensors may also include a position sensor that acquires position information on the vehicle. The example of
The system 700 performs the processes illustrated in
In step 301 of
In step 302 of
In the example of
The map image may be received from an external computer through a communication network, or may be stored in advance in the storage means 702 of the system 700. The map image may be also directly acquired as an image, or may be acquired as an image format after information acquired in a format other than an image is converted. The conversion may be executed with reference to other information. For example, the system 700 may acquire map information in a two-dimensional format and generate a pseudo-three-dimensional map image as illustrated in
In step 303 of
The first area is likely to include an object directly related to safety for the moving vehicle 200, and can be called a high-risk area. The first area is also likely to include an object moving with respect to the road surface, and can also be called a dynamic area. In contrast, the second area is unlikely to include an object directly related to safety for the moving vehicle 200, and can be called a low-risk area. The second area is also unlikely to include an object moving with respect to the road surface, and can also be called a static area.
Hereinafter, although in the present embodiment, the first area is referred to as the “high-risk area” and the second area is referred to as the “low-risk area”, for convenience of explanation, names of these areas are not essential to the present invention.
In step 304 of
The first transmission determination function may be executed, for example, based on an effective communication rate of the communication network. More specifically, when the effective communication rate of the communication network to the remote support system is equal to or higher than a predetermined threshold value, it is determined that data related to the high-risk area should be transmitted, and otherwise it is determined that the data should not be transmitted. According to such criteria, the amount of data to be communicated can be reduced. In particular, when the effective communication rate is low, communication capacity can be saved for other more important data.
The effective communication rate may be a value called “bandwidth”, “channel capacity”, “transmission line capacity”, “transmission delay”, “network capacity”, “network load”, or the like. A method for measuring the effective communication rate can be appropriately designed by those skilled in the art based on known techniques and the like.
The first transmission determination function may be executed based on the number of objects detected in the high-risk area, which is, for example, determined in step 306 or 307. In that case, the first determination function may be executed after step 307, but before step 309. More specifically, when the number of objects exceeding a predetermined threshold value belongs to the high-risk area, it is determined that the data related to the high-risk area should be transmitted, and otherwise it is determined that the data should not be transmitted. According to such criteria, when the number of objects exceeding a limit that can be processed by the system 700 itself is detected, assistance of the remote assistance system can be appropriately requested.
The first transmission determination function may be executed based on a comparison of computational capability between the system 700 and the remote assistance system. For example, the function may be executed based on a relative value representing the computational capability of the system 700 with respect to the remote assistance system. Such a relative value can be determined using a function, which may be, for example, a simple division or subtraction, the function including a value representing the computational capability of the remote assistance system and a value representing the computational capability of the system 700. For example, when the system 700 has a failure, the computational capability of the system 700 may be evaluated lower.
As a more specific example, when a relative value representing the computational capability of the system 700 is equal to or more than a predetermined threshold value, it is determined that the data related to the high-risk area should not be transmitted, and otherwise it is determined that the data should be transmitted. According to such criteria, the amount of data to be communicated can be reduced. Only when determination capability of the system 700 itself is insufficient, the assistance of the remote assistance system can be efficiently requested.
The first transmission determination function may be executed by combining the plurality of criteria described above.
In step 304 of
The second transmission determination function may be executed, for example, based on accuracy of a position of the system 700. In the present embodiment, the position of the system 700 can be regarded as the same as the position of the vehicle 200. For example, the system 700 can acquire or calculate the position of the system 700 and accuracy of the position (i.e., the position of the vehicle 200 and accuracy of the position) based on data detected by the GPS and the INS 207. When the accuracy is equal to or more than a predetermined threshold value, it is determined that data related to the low-risk area should not be transmitted, and otherwise it is determined that the data should be transmitted.
Here, the low-risk area is likely to include many static features related to the map image, and thus is likely to be useful for precise determination of the position of the vehicle 200 or the system 700. Thus, according to such criteria, assistance of the remote assistance system can be appropriately requested only when it is difficult for the system 700 to identify its own position independently.
In the present embodiment, the system 700 may not necessarily operate in step 304 according to
The conditions referred to in the first transmission determination function and the second transmission determination function may include an effective communication rate of the communication network, the number of detected objects, a computational capability value of the remote assistance system, a computational capability value of the system 700, accuracy of a position of the system 700, and moving speed of the system 700 (i.e., traveling speed of the vehicle 200), for example. Additionally, various combination patterns of these conditions may be defined, and the storage means 702 may store a determination table in which whether data related to the high-risk area should be transmitted is associated with whether data related to the low-risk area should be transmitted, for each of the patterns. On the basis of these conditions, the system 700 can perform the first transmission determination function and the second transmission determination function with reference to the determination table.
In step 305 or step 306 of
In the example of
As a more specific example, when the first front camera 203 detects an image as illustrated in
Surrounding objects may be detected based on other data. For example, the objects may be detected based on an image detected by another camera, or may be detected based on data detected by a sensor other than the camera, such as a LIDAR sensor, a RADAR sensor, an ultrasonic sensor, or an audio sensor.
In step 306 or 307 of
In step 306 or 307 of
In this determination, when a part of an object belongs to one area and another part of the object does not belong to the one area (e.g., when the object exists across high-risk and low-risk areas), processing of the determination can be appropriately designed by those skilled in the art. For example, the object may be determined based on its center of gravity on an image.
In step 308 of
For example, an object with a short distance may be determined to have a small data compression ratio (i.e., a large amount of data after compression or a small amount of information loss), and an object with a large distance may be determined to have a large data compression ratio (i.e., a small amount of data after compression or a large amount of information loss). In the present embodiment, the system 700 may not necessarily operate in step 308 according to
This causes an object that is more important in determining operation of the system 700 or vehicle 200, i.e., an object that is closer to the system 700 or vehicle 200, to have a small amount of loss by using a larger amount of data. As a result, more secure operation of the vehicle 200 is likely to be able to be determined. In contrast, for an object that is less important in determining the operation of the system 700 or vehicle 200, i.e., an object that is farther from the system 700 or vehicle 200, data is compressed more strongly to reduce the amount of the data, so that communication capacity can be saved.
The compression ratio determination function does not need to be executed based only on a distance to an object, and other criteria may be used in combination. For example, the function may be executed based further on a type (class) of each object or a behavior of each object. As a more specific example, a compression ratio may be reduced when the object is a pedestrian, and may be increased when the object is a vehicle. In particular, for a vehicle, the amount of data after compression may be zero or almost zero, or image information may be discarded to leave only convex hull information. This enables assistance of the remote assistance system to be appropriately requested by reducing the amount of information on a vehicle that frequently appears in an image of an in-vehicle camera, and leaving more information on a pedestrian that appears less frequently.
Alternatively, when an object is approaching the vehicle 200 (or system 700), a compression ratio may be reduced, and when an object is moving away from the vehicle 200 (or system 700), a compression ratio may be increased. This enables assistance of the remote assistance system to be appropriately requested by leaving more information on an object that is important for determining operation of the vehicle 200.
Alternatively, the compression ratio determination function may be further executed based on an effective communication rate of the communication network. As a more specific example, when the effective communication rate is equal to or higher than a predetermined threshold value, the compression ratio may be reduced, and otherwise the compression ratio may be increased. This enables communication with an appropriate amount of data to be achieved according to available communication capacity.
For an area determined not to transmit data, execution of the compression ratio determination function may be eliminated. For example, when it is determined not to transmit data related to the high-risk area, a data compression ratio of an object belonging to the high-risk area does not need to be determined.
In step 309 of
This process may be eliminated for the area determined not to transmit data. For example, when it is determined not to transmit data related to the high-risk area, compression data related to an object belonging to the high-risk area does not need to be generated.
In step 309 of
These compressed data are transmitted to, for example, the remote assistance system. As a modification, these compressed data may be transmitted to a computer system other than the remote assistance system. For example, the data may be transmitted to another system being mounted on a vehicle other than the vehicle 200 and having the same configuration as the system 700. In that case, the other system may function as a relay base between the system 700 and the remote assistance system. Additionally, in that case, the other system may function as a relay base between a plurality of systems including the system 700 and the remote assistance system. This enables reducing the number of systems that directly communicate with the remote assistance system, and reducing congestion of communication in the remote assistance system.
Although not illustrated in
In step 310 of
In step 310 of
| Number | Date | Country | Kind |
|---|---|---|---|
| 2019-051272 | Mar 2019 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2019/050011 | 12/20/2019 | WO | 00 |
