Patent Application
20210287023
The present disclosure relates to traffic signal identification and, in particular, to using bulb mask representation to identify traffic signal devices and the states of various elements of traffic signal devices.
Traffic signal devices are critical to safe driving. They signal when it is the safe, legal, and appropriate time for vehicles to pass or enter certain intersections or other regions. For this reason, autonomous vehicles require the ability to accurately detect the various signal elements of the traffic signal devices and the states of each of those signal elements. Once determined, this information is used to accurately instruct the autonomous vehicles on how to react to a traffic signal device.
Various techniques are available for identifying traffic signal devices. One such technique is the use of one or more deep learning methods. Deep learning techniques and methods have the advantage of being largely data-driven and able to integrate information from a wide range of example images into a single classifier. While many architectures of deep learning models follow a common pattern of successive layers that reduce the dimensionality of an input image, the methods by which these models represent their inputs and outputs vary substantially.
Given that the detection of the state of a traffic light is ultimately a classification problem, once a traffic light is detected it is known that the light's state may be determined by using representations typically used for image classification, in which each possible state is encoded by one position in a one-hot vector. For example, a classifier that classifies images as containing a dog or a cat might represent its output as [probability_of_cat, probability_of_dog]. Analogously, a traffic light classifier might represent its outputs as [probability_red, probability_yellow, probability_green].
This approach works for most traffic lights, but becomes more complicated in light of traffic light faces that might have multiple bulb types of the same color (for example, traffic light faces with both arrows and circles, or multiple types of arrows on the same face). These types of traffic signal devices are common and, therefore, the ability to identify the various elements on these traffic signal devices and the states of each of these various elements is paramount.
In these scenarios, the output vector would need to be expanded to include every type and color combination, yet this still does not fully solve the problem due to the oftentimes complex nature of the faces of traffic signal devices. For example, some traffic light faces may have multiple bulbs of the same type and color that are differentiated only by their positions on the face of the traffic signal device. One example of this is the HAWK beacon, in which there are two red circle bulbs at the top of the face that flash in an alternating pattern.
Therefore, for at least these reasons, a better method of identifying and classifying various forms and states of traffic signal devices is needed.
According to an aspect of the present disclosure, a method for representing a traffic signal device is provided. The method includes receiving, by a computer vision system of a vehicle, a digital image of a traffic signal device that includes one or more traffic signal elements. The method further includes, by a processor, representing the traffic signal device as a raster image, each traffic signal element of the traffic signal device being represented by a mask corresponding to a location of the traffic signal element on the traffic signal device, representing each mask in a channel in the raster image, in which the channel representing each mask corresponds to one or more features of each of the one or more traffic signal elements, providing the raster image as an input to a neural network to classify a state for each of the one or more traffic signal elements, and receiving, from the neural network, a classified raster image, in which the classified raster image includes a plurality of masks, each mask representing a state of one of the one or more traffic signal elements.
According to various embodiments, representing the traffic signal device as a raster image includes using map data to directly generate the raster image.
According to various embodiments, representing the traffic signal device as a raster image includes generating a digital image from map data.
According to various embodiments, each channel is an image color channel.
According to various embodiments, each image color channel corresponds to a unique color.
According to various embodiments, each traffic signal element of the traffic signal device corresponds to a designated light fixture configured to transmit traffic instructions to one or more drivers.
According to various embodiments, the one or more features of each of the one or more traffic signal elements includes one or more of the following: a green light; a yellow light; a red light; a circular light; a left arrow light; a right arrow light; a forward arrow light; a light having an arrow in any direction; a flashing green light, a flashing yellow light; a flashing red light; a U-turn light; a bicycle light; and an X-light.
According to various embodiments, each mask corresponds to a discrete region of pixels in the raster image.
According to various embodiments, each discrete region of pixels is rectangular in shape. The discrete region of pixels may be any suitable shape.
According to various embodiments, the method further includes identifying a face of the traffic signal device, each traffic signal element being located within the face of the traffic signal device.
According to various embodiments, the method further includes generating a confidence value that a traffic signal element of a traffic signal device correlates to a state, and if the confidence value is greater than a threshold value, determining that the traffic signal element is a state.
According to various embodiments, the method further includes using a position and shape of each mask in the raster image to identify an angle of the traffic signal device. For enumerable fields, the information may be encoded using the color channel in the mask, as well as a binary value (0 for false, 1 for true, for example). For continuous fields, the information may be encoded in a particular color channel, where the value of the mask indicates the value of the continuous field. For example, the red channel may have values between 0 and 200 meters.
According to another aspect of the present disclosure, a system for representing a traffic signal device is provided. The system includes a vehicle, a computer vision system of the vehicle, configured to receive a digital image of a traffic signal device that includes one or more traffic signal elements, and a transceiver configured to send and receive digital information. The system further includes a processor configured to represent the traffic signal device as a raster image in which each traffic signal element of the traffic signal device is represented by a mask corresponding to a location of the traffic signal element on the traffic signal device, to represent each mask in a channel in the raster image, in which the channel representing each mask corresponds to one or more features of each of the traffic signal elements, provide, using the transceiver, the raster image as an input to a neural network to classify a state for each traffic signal element, and receive, from the neural network, using the transceiver, a classified raster image, in which the classified raster image includes a plurality of masks, each of which represents a state of one of the traffic signal elements.
According to various embodiments, the processor is further configured to represent the traffic signal device as a raster image using map data to directly generate the raster image.
According to various embodiments, the processor is further configured to represent the traffic signal device as a raster image by generating a digital raster image from map data.
According to various embodiments, each traffic signal element of the traffic signal device corresponds to a designated light fixture configured to transmit traffic instructions to one or more drivers.
According to various embodiments, the one or more features of the traffic signal elements includes one or more of the following: a green light; a yellow light; a red light; a circular light; a left arrow light; a right arrow light; a forward arrow light; a light having an arrow in any direction; a flashing green light, a flashing yellow light; a flashing red light; a U-turn light; a bicycle light; and an X-light.
According to various embodiments, each mask corresponds to a discrete region of pixels in the raster image.
According to various embodiments, the processor is further configured to identify a face of the traffic signal device, each traffic signal element being located within the face of the traffic signal device.
According to yet another aspect of the present disclosure, a system for representing a traffic signal device is provided. The system includes a vehicle including a computer vision system configured to receive a digital image of a traffic signal device that includes one or more traffic signal elements. The vehicle further includes a computer-readable storage medium comprising one or more programming instructions that, when executed, cause the vehicle to represent the traffic signal device as a raster image, wherein each traffic signal element of the traffic signal device is represented by a mask corresponding to a location of the traffic signal element on the traffic signal device, represent each mask in a channel in the raster image, in which the channel representing each mask corresponds to one or more features of each of the one or more traffic signal elements, provide the raster image as an input to a neural network to classify a state for each of the one or more traffic signal elements, and receive, from the neural network, a classified raster image, in which the classified raster image includes a plurality of masks, wherein each mask represents a state of one of the one or more traffic signal elements.
In this document, when terms such as “first” and “second” are used to modify a noun, such use is simply intended to distinguish one item from another, and is not intended to require a sequential order unless specifically stated. The term “approximately,” when used in connection with a numeric value, is intended to include values that are close to, but not exactly, the number. For example, in some embodiments, the term “approximately” may include values that are within +/−10 percent of the value.
In this document: (i) the term “comprising” means “including, but not limited to”; the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise; and (iii) unless defined otherwise, all technical and scientific terms used in this document have the same meanings as commonly understood by one of ordinary skill in the art. Also, terms such as “top” and “bottom”, “above” and “below”, and other terms describing position are intended to have their relative meanings rather than their absolute meanings with respect to ground. For example, one structure may be “above” a second structure if the two structures are side by side and the first structure appears to cover the second structure from the point of view of a viewer (i.e., the viewer could be closer to the first structure).
An “electronic device” or a “computing device” refers to a device that includes a processor and memory. Each device may have its own processor and/or memory, or the processor and/or memory may be shared with other devices as in a virtual machine or container arrangement. The memory will contain or receive programming instructions that, when executed by the processor, cause the electronic device to perform one or more operations according to the programming instructions. Examples of electronic devices include personal computers, servers, kiosks, mainframes, virtual machines, containers, gaming systems, televisions, and mobile electronic devices such as smartphones, personal digital assistants, cameras, tablet computers, laptop computers, media players and the like. In a client-server arrangement, the client device and the server are electronic devices, in which the server contains instructions and/or data that the client device accesses via one or more communications links in one or more communications networks. The server may be a single device or a collection of devices that are distributed but via which processing devices and/or memory are shared. In a virtual machine arrangement, a server may be an electronic device, and each virtual machine or container may also be considered to be an electronic device. In the discussion below, a client device, server device, virtual machine or container may be referred to simply as a “device” for brevity.
In this document, the terms “memory,” “memory device,” “data store,” “data storage facility” and the like each refer to a non-transitory device on which computer-readable data, programming instructions or both are stored. Except where specifically stated otherwise, the terms “memory,” “memory device,” “data store,” “data storage facility” and the like are intended to include single device embodiments, embodiments in which multiple memory devices together or collectively store a set of data or instructions, as well as individual sectors within such devices.
In this document, the terms “processor” and “processing device” refer to a hardware component of an electronic device that is configured to execute programming instructions. Except where specifically stated otherwise, the singular term “processor” or “processing device” is intended to include both single-processing device embodiments and embodiments in which multiple processing devices together or collectively perform a process.
In this document, “electronic communication” refers to the transmission of data via one or more signals between two or more electronic devices, whether through a wired or wireless network, and whether directly or indirectly via one or more intermediary devices. Devices are “communicatively connected” if the devices are able to send and/or receive data via electronic communication.
Referring now to
According to various embodiments, the system 100 includes a vehicle 105. According to various embodiments, the vehicle 105 is traveling on a road 110. It is noted, however, that any suitable path may be implemented.
The vehicle 105 may include a computer vision system 115 configured to receive a digital image of a traffic signal device 130. The computer vision system 115 may include one or more cameras for capturing digital images of various features of the environment in which the vehicle 105 is traveling. Such features may include one or more traffic signal devices 130. According to various embodiments, the digital image is a raster image, such as that shown in
The signal elements 135 are represented in
The vehicle 105 may further include a transceiver 120 configured to send and receive digital information from a remote server 145 via a wired and/or wireless connection such as, for example, through the cloud 140, wherein the vehicle 105 and the remote server 145 are in electronic communication with each other. The vehicle 105 may further include a processor 125. The processor 125 may be configured to represent the traffic signal device 130 as a raster image (such as that shown in
As shown in
In order for an autonomous vehicle 105 to ascertain the signal elements 135 of a traffic signal device 130, such as that shown in
According to various embodiments, in order to classify the signal elements 135 of the traffic signal device 130, the traffic signal device 130 is represented as a simplified two-dimensional image with at least one or more image color channels, where the signal elements 135 of the traffic signal device 135 correspond to discrete regions of pixels within the two-dimensional image. According to various embodiments, each image color channel corresponds to a unique color. Therefore, in the two-dimensional image, the signal elements 135 that are separated in space on the face of the actual traffic signal device 130 are analogously separated in space on the two-dimensional image representation of the traffic light face. This spatial separation in the two-dimensional raster image signifies that the traffic light classifier system 100 is able to represent any number of signal elements 135 with any number of color and type combinations.
According to various embodiments, the processor 125 may further be configured to represent each mask in a channel in the raster image, in which the channel representing each mask corresponds to one or more features of each of the traffic signal elements 135. Such raster images are shown in
In addition to the spatial location, the raster image can indicate the colors and types of each signal element 135. It does so by representing different colors and types of signal element 135 as different channels. For example, green, yellow, and red lights may correlate to channels 0, 1 and 2, and circular lights, straight arrow lights, right arrow lights, and left arrow lights may correspond to channels 3, 4, 5, and 6. It is noted, however, that the features of the signal elements 135 and the colors, shapes, and channels used may be of any suitable type and number. According to various embodiments, one or more scalar values such as, for example, the angle of the face of the traffic signal device 130, may be represented as one or more scalar values in a particular channel.
As shown in
As shown in
According to various embodiments, the processor 125, using the transceiver 120, is further configured to provide one or more raster images (for example, those shown in
In
The state image indicates which signal elements 135 are lit. It performs this by, for example, filling in regions corresponding to the lit signal elements 135 with ones and the other regions of the face of the traffic signal device 130 with zeros. It is noted, however, that other means of determining which signal elements 135 are lit may additionally, or alternatively, be used. According to various embodiments, the groupings of ones may indicate a confidence that a signal element 135 is lit. According to various embodiments, the values may be intermediate floating point values (for example, 0.5) if there is uncertainty concerning which bulbs are lit. The state image is used to generate the classified raster image, such as that shown in
Referring now to
While driving, a vehicle may come across a traffic signal device. At 705, using a processor, a face of the traffic signal device is identified. It is noted that the processor may perform one or more of the steps described herein in method 700. At 710, a digital image of a traffic signal device is received/captured by a computer vision system of a vehicle. According to various embodiments, the digital image is a real world image. According to various embodiments, the “real world image” is a digitally captured photograph of a view from the perspective of the vehicle. The computer vision system may include, for example, a camera. The traffic signal device includes one or more traffic signal elements on the face of the traffic signal device, wherein each traffic signal element of the traffic signal device corresponds to a designated light fixture configured to transmit traffic instructions to one or more drivers.
At 715, the face of the traffic signal device is represented as a raster image, wherein each traffic signal element of the traffic signal device is represented by a mask corresponding to a location of the traffic signal element on the traffic signal device, forming a template image. Once the template image is formed, each mask, at 720, is represented in a channel in the raster image, in which the channel representing each mask corresponds to one or more features of each of the one or more traffic signal elements. Optionally, the angle of the traffic signal device represented in the mask image may, at 725, be identified based on a position and shape of each mask on one or more of the raster images.
Using the captured image of the traffic signal device, at 730, the image of the traffic signal device and the rasterized images are provided as an input to a neural network to classify a state for each of the one or more traffic signal elements and generate a classified raster image. The neural network, at 735, analyzes the input to determine a state of each of the traffic signal elements and, at 740, generates a confidence score indicating a confidence that a particular traffic signal element has a specified state. According to various embodiments, a confidence score is placed into the classified raster image. At 745, it is determined whether the confidence score is above a threshold. If the confidence score is equal to or above the threshold, the traffic signal element, at 750, is determined to have a specified state. If the confidence score is below the threshold, the traffic signal element, at 755, is determined to not have the specified state.
After identifying the state of each of the traffic signal elements, the classified raster image, at 760, is generated. The classified raster image includes a one or more masks, wherein each mask represents a state of one or more one or more traffic signal elements. Once generated, the processor, at 765, receives the classified raster image.
A bus 900 serves as the main information highway interconnecting the other illustrated components of the hardware. CPU 905 is the central processing unit of the system, performing calculations and logic operations required to execute a program. CPU 905, alone or in conjunction with one or more of the other elements disclosed in
Program instructions, software or interactive modules for providing the interface and performing any querying or analysis associated with one or more data sets may be stored in the computer-readable storage media 910. Optionally, the program instructions may be stored on a tangible, non-transitory computer-readable medium such as a compact disk, a digital disk, flash memory, a memory card, a USB drive, an optical disc storage medium and/or other recording medium.
An optional display interface 915 may permit information from the bus 900 to be displayed on the display 920 in audio, visual, graphic or alphanumeric format. Communication with external devices may occur using various communication ports 925. A communication port 925 may be attached to a communications network, such as the Internet or an intranet. In various embodiments, communication with external devices may occur via one or more short range communication protocols.
The hardware may also include an interface 930, such as graphical user interface, which allows for receipt of data from input devices such as a keyboard or other input device 935 such as a mouse, a joystick, a touch screen, a remote control, a pointing device, a video input device and/or an audio input device.
The features and functions described above, as well as alternatives, may be combined into many other different systems or applications. Various alternatives, modifications, variations or improvements may be made by those skilled in the art, each of which is also intended to be encompassed by the disclosed embodiments.
