The present invention relates to an image processing device mounted on a vehicle.
Vehicle-mounted image processing devices recognize an object from an image captured by an image sensor, and, based on that recognition result, control the vehicle body and issue various warnings to realize an active safety function.
However, it is not just observation objects of interest for the active safety of the vehicle, e.g., people, vehicles, etc., that are found in images captured by an image sensor. Accordingly, it is necessary to execute a process of extracting image data with respect to all regions that appear to be objects, and to execute a recognition process for categorizing the extracted image data into people, vehicles, etc.
Then, as a result of recognition, it is determined what state the people or vehicles are in, and if a collision appears likely, decisions are made such as whether or not to execute such control as operating the brakes, avoiding a collision through a steering operation, etc., or whether a warning should be issued although the likelihood of a collision is low. There are infinite scenes that call for such decisions. As such, all such scenes were to be accommodated, the processing loads of the extraction process, recognition process, and determination process would become enormous for the image processing device.
As a method of mitigating the processing load of an image processing device, there is a method where scenes that require determination are restricted. One example is a method that restricts itself to scenes where an object is present in proximity to the host vehicle, and recognition and determination are executed only for the object closest to the host vehicle.
On the other hand, there are also methods that handle processing loads using a plurality of CPUs. Many multi-processor image processing devices divide the screen and allocate processors to the respective divided regions to even out the processing loads. In addition, many multi-core image processing devices realize data parallel processing, such as a filtering process, in which the same process is executed in parallel with respect to given pixels or regions (see, for example, Patent Documents 1 and 2).
However, with a method that restricts the scenes for which a determination is required, there is a risk that dangerous situations might be tolerated. By way of example, with a method in which scenes are restricted to those in which objects are present in proximity to the host vehicle, and in which recognition and determination are executed only for the object closest to the host vehicle, in a scene where the host vehicle is traveling a road that has no sidewalk, a person is present in proximity, and there is a vehicle that is traveling in the opposite direction at an extremely high speed, recognition and determination would be performed only for the person who is the object closest to the host vehicle. However, in such a scene, it would be preferable to give priority to and recognize the vehicle traveling in the opposite direction at an extremely high speed.
Thus, since there are various types of objects to be sensed, e.g., people, vehicles, etc., and there are an infinite number of scenes in which these are combined, there was a problem in that when processing is performed, for example, only for the object closest to the host vehicle, and so forth, scenes that should be avoided in the real world could not be covered comprehensively.
In addition, in the case of a multi-processor system in which CPUs or cores are allocated to the respective divided regions of the screen, depending on the circumstances, regions with a low processing load occur, in which case the processing capabilities of the processors allocated to those regions would not be utilized effectively, and thus be wasted. Further, in the case of data parallel processing where the same process is executed in parallel with respect to given pixels or regions, efficiency is poor, and processing cannot be sped up.
The present invention is made in view of the points mentioned above, and an object thereof is to provide a vehicle-mounted image processing device that is able to make a plurality of processing units appropriately perform a process that suits a scene in which various people, vehicles, etc., are combined.
A vehicle-mounted image processing device of the present invention that solves the above-mentioned problems is characterized in that, based on processing results that have been image processed by a plurality of processing units, tag information indicating the type and priority of the image processing to be executed next by each processing unit is generated, and whether or not to make each processing unit perform image processing is determined using the tag information generated during previous image processing.
An image processing device of the present invention generates, based on processing results that have been image processed by a plurality of processing units, tag information indicating the type and priority of image processing to be executed next by each processing unit, and determines, using the tag information generated during previous image processing, whether or not to make each processing unit perform image processing. Accordingly, it is possible to make a plurality of processing units appropriately perform processing in order of descending priority. Thus, scenes to be avoided in the real world may be covered comprehensively without any omission, and safety while the vehicle is traveling may be improved dramatically. The present specification incorporates the contents of the specification and/or drawings of JP Patent Application No. 2010-027747 from which the present application claims priority.
10 Line buffer
20 Region extraction means
30 Image buffer
40 Processing unit
50 Determination means
60 Storage device
62 Current priority
70 Controller (tag information generation means)
80 Command buffer
90 Program
100 Communication channel
110 Packet
112 Processing result
114 Region information
130 Status management table
140 Program management table
150 Priority management table
Next, embodiments of the present invention are described using the drawings
As shown in
The line buffer 10 is provided in this example because a typical image sensor used as a vehicle-mounted camera is structured to output image data line by line. A video input (Vin) through which signals from the vehicle-mounted camera are inputted is connected to the line buffer 10. By scanning line by line in the perpendicular direction, the line buffer 10 is able to generate one screen's worth of image data. This perpendicular scanning time is referred to as the frame period.
Within the frame period for generating one screen's worth of image data, the region extraction means 20 divides the image data into desired processing regions at least in units of pixels, lines, stripes, or blocks, and extracts the image data of those processing regions. In synchrony with a perpendicular synchronization signal of the vehicle-mounted camera, etc., the region extraction means 20 reads one line's worth of image data from the line buffer 10 and repeats a process of acquiring the necessary pixel data, thereby extracting the image data of the processing regions.
The image buffer 30 stores the image data extracted at the region extraction means 20. The image buffer 30 is capable of storing a plurality of image data so as to be able to accommodate cases where there are more than one processing regions.
The plurality of processing units 40 each comprise a processor, such as a CPU, a core, etc. Each processing unit 40 performs image processing based on image data extracted by the region extraction means 20. Each processing unit 40 comprises a determination means 50, which determines whether or not to execute image processing, and a storage device 60.
Using the tag information generated by the controller 70 during previous image processing, the determination means 50 determines whether or not to make each processing unit 40 perform image processing. The storage device 60 is used for storing a program 90 and image data. However, the present invention is by no means limited to this configuration.
The controller 70 comprises a means for managing the relationship among processing result information of each processing unit 40, image data of processing regions, types of processes, and information indicating process priority. Based on the processing results of image processing outputted by the processing units 40, the controller 70 (tag information generation means) generates tag information indicating the type and priority of the image processing to be executed next by each processing unit 40.
The command buffer 80 manages the types of processes to be executed at the processing units 40, and the programs 90 corresponding to those process types. It is able to transmit the programs 90 to the processing units 40 based on the tag information generated by the controller 70 during previous image processing.
The processing unit 40 executes image processing, generates a processing result 112 thereof and region information 114 of a processing region corresponding to the processing result, and transmits them to the controller 70 as a packet 110 (S10).
Upon acquiring the packet 110, the controller 70 searches a table 130 (
By way of example, in a case where the processing unit 40 executes, as image processing, a three-dimensional object sensing process with respect to the entire screen and is unable to sense any three-dimensional objects, the processing unit 40 puts “three-dimensional object not sensed” information and “region information” indicating the entire screen as processing results together into a packet and transmits it to the controller 70.
The controller 70 receives this packet via the communication channel 100. The controller 70 manages the table 130, which defines the type of process to be executed next with respect to the processing results and the priority of that process, and searches that table 130. The table 130 is comparable to a state transition table.
In the present example, as types of processes for “three-dimensional object not sensed,” there are stored “long-range sensing process” and “short-range sensing process,” where the priority of “long-range sensing process” is set high and the priority of “short-range sensing process” is set low. In accordance with this table 130, the controller 70 generates two types of information, namely, information that causes the long-range sensing process, which has high priority, to be executed, and information that causes the short-range sensing process, which has low priority, to be executed. Each of the types of information above is put together into a packet and transmitted.
The region extraction means 20 compares the Y-coordinate with the order of the synchronization signal of the line buffer 10 (S100), and, if they are the same, imports the image data of the line buffer 10 (S110). It then extracts the image data of the dot that coincides with the X-coordinate (S120), and transfers the extracted image data to the image buffer 30 along with region information (S130).
The region extraction means 20 compares the Y-coordinate with the order of the synchronization signal of the line buffer 10 (S200). If they are the same, it copies the image data of the line buffer 10 as is (S210). Further, it transfers the copied image data to the image buffer 30 along with region information (S220).
In the case of blocks, region information is indicated by a combination of a starting point and end point for the X-coordinate and a starting point and end point for the Y-coordinate. The region extraction means 20 compares the Y-coordinate with the order of the synchronization signal of the line buffer 10, and determines if the order of the synchronization signal of the line buffer 10 exceeds the starting point for the Y-coordinate (S340). If it falls between the starting point and end point for the Y-coordinate, the image data of the line buffer 10 is imported (S350). Further, image data of from the starting point to the end point for the X-coordinate is cut out (S360) and additionally written (S370). If it is the same as the end point for the Y-coordinate (YES in S380), the copied and additionally written image data and the region information are transferred to the image buffer 30 together.
As shown in
The table 150 manages the types of processes to be executed next with respect to the image processing results, and the priorities of these processes. By way of example, for the processing result “three-dimensional object sensed,” two types of processes, namely “human recognition process” and “vehicle recognition process,” are defined for the processes to be executed next. “Human recognition process” and “vehicle recognition process” have the same priority.
In addition, for the processing result “vehicle recognized,” two types of processes, namely “avoidance control process” and “collision warning process,” are defined for the processes to be executed next. However, between avoidance and warning, the former has greater importance. Accordingly, “avoidance control process” has higher priority than “collision warning process.” In the present example, priority is so defined as to be greater the smaller the numerical value. Although they are represented as the table 150 comparable to a state transition table in the present example, this is by no means limiting.
In the present example, as shown in
In this case, by dividing the processing region and autonomously distributing processing among the plurality of processing units 40 based on the total number of the processing units 40 and on priority, it is possible to prevent the processing load from becoming concentrated at just one processing unit 40.
By way of example, as shown in
Then, as the recognition object 160 approaches the host vehicle, it occupies a larger region in the image 121 as shown in
On the other hand, if the vehicle speed of the host vehicle is low, recognition of objects nearby must be enabled. In such low-speed ranges, it is necessary to improve precision for near object sensing. Accordingly, in a low-speed range 184, as shown in
By thus setting processing regions in accordance with the speed range corresponding to the behavior of the host vehicle, it is possible to accommodate various situations of objects. In the present example, speed was handled as information representing the behavior of the host vehicle. However, by setting processing regions in a flexible manner in accordance with, by way of example, the steering angle or steering speed of the steering wheel, it becomes possible to accommodate various situations at curves and intersections.
On the other hand, if, as a result of image processing, an object ahead is sensed as a recognition object 160 as shown in
The processing unit 40 receives from the controller 70 a packet containing tag information generated during previous image processing, and acquires information on the type and priority of the process (S500). Then, the determination means 50 makes a determination by comparing the current priority and the received priority (S510). In the present example, the determination means 50 simply determines whether the received priority is higher or lower than the current priority, and if higher, updates the current priority with that priority (S520).
The processing unit 40 transmits the process type to the command buffer 80 (S530), and receives from the command buffer 80 the program 90 corresponding to the process type (S540). Further, if the received priority is low, the current priority is reset (S522).
The processing unit 40 generates a priority through the program 90 and updates the current priority (S640). If it is possible to uniquely determine priority from the design stage, by clearly generating a priority in a program as in the present example, reliable priority control may be effected.
The processing unit 40 receives from the controller 70 a packet containing tag information generated during previous image processing, and acquires information on the type and priority of the process from the tag information (S700). The processing unit 40 transmits the process type to the command buffer 80 (S710), and acquires from the command buffer 80 the program 90 corresponding to the process type (S720). The processing unit 40 executes the program 90 (S730), and checks whether or not an urgency level has been generated (S740).
If an urgency level has been generated, the urgency level is transmitted to the command buffer, and the program 90 corresponding to the urgency level is acquired from the buffer 80 (S750). The processing unit 40 executes the acquired urgent program 90 (S760). By employing such a flow and promptly executing a process of high urgency without going through the controller 70, real-time properties may be improved.
As shown in
As shown in
Before receiving the image data 228, each processing unit 40 receives a program from the command buffer 80, pre-fetches the program, and subsequently receives the image data, thereby beginning image processing.
By way of example, at the time at which the importing of image data from the line buffer 10 is begun, the program 90 corresponding to the process type may be downloaded from the command buffer 80, and the processing unit 40 is able to begin execution earlier as compared to downloading after the importing of image data has been completed.
Taking the packet 220 shown in
The processing unit 40 receives from the controller 70 the packet containing tag information generated during previous image processing, and acquires, from the tag information, information on the type and priority of the process and the image ID (S805). It is then determined whether or not the image ID is valid (S810), and if the image ID is invalid, the process type is transmitted to the command buffer 80 (S820).
Upon receiving the process type (S830), the command buffer 80 searches for the program 90 corresponding to that process type, and transmits the program 90 to the processing unit 40 (S840). The processing unit 40 receives the program 90 (S850), and if the image ID is valid, transmits the image ID to the image buffer 30 (S860).
Upon receiving the image ID (S870), the image buffer 30 transmits to the processing unit 40 the image data corresponding to the image ID (S880). Upon receiving the image data (S890), the processing unit 40 executes the program 90 to process the image data (S900). It is noted that the region extraction means 20 transmits the type and priority of the process and a valid image ID upon completing the generation of image data for the image region information.
In the present example, the controller 70 manages the total number of processing units 40 and the number of processing units 40 that are executing a process. The controller 70 receives a processing result from the processing units 40 (S900). Then, as in the example in
From the total number of processing units 40, the number of processing units 40 executing a process, and the number of processing units 40 that are idle, the controller 70 calculates the number of processing units 40 that are in a process waiting state (S920), and transmits packets in a quantity corresponding to the number of processing units 40 that are in a process waiting state (S930). These packets specify, by way of example, a sleep program for the process type. Upon receiving a sleep program packet from the controller 70, the processing unit 40 changes the operation mode of the processing unit 40 from the normal mode to a sleep mode that consumes less power. Accordingly, the power consumed by the processing units 40 in a process waiting state may be reduced.
According to the image processing device discussed above, tag information representing the type and priority of the process to be executed next by each processing unit 40 is generated based on the processing result of each processing unit 40, and whether or not each processing unit 40 is to perform image processing is determined using the tag information generated during previous image processing. Consequently, it is possible to make a plurality of processing units appropriately perform processes in order of descending priority. Accordingly, scenes to be avoided in the real world may be covered comprehensively, enabling a dramatic improvement in safety while the vehicle is traveling.
Further, with respect to a processing region for which image processing needs to be performed with greater precision, the processing region may be divided and a processing unit may be assigned to each. Thus, processing may be distributed among a plurality of processing units 40 with priority, and the processing load may be prevented from becoming concentrated at a single processing unit 40. Thus, image processing may be sped up, warnings and automatic brake control, etc., based on image processing results may be performed promptly, and safety while the vehicle is traveling may be improved dramatically.
Number | Date | Country | Kind |
---|---|---|---|
2010027747 | Feb 2010 | JP | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/JP2010/063757 | 8/13/2010 | WO | 00 | 8/8/2012 |