The present disclosure relates generally to the automotive field. More particularly, the present disclosure relates to a system and method for providing in-vehicle emergency vehicle detection and positional alerts.
An emergency vehicle is defined generally as a police car, a fire truck, an ambulance, or the like. When responding to an emergency, these emergency vehicles are likely moving (potentially at a high rate of speed) with flashing lights and broadcasting a siren. By law, when a driver is approached or passed by an emergency vehicle, the driver must move over, slow down, stop, and/or otherwise give way and provide the emergency vehicle with safe passage. However, sometimes the driver fails to promptly notice the emergency vehicle, or fails to properly judge the emergency vehicle's position and direction of travel. This can result due to driver inattention, poor visibility, ambient noise, etc. The outcome may be unintended interference with the emergency vehicle, slowing its response to an emergency, or, in a worst case scenario, a traffic incident involving the ego vehicle and the emergency vehicle.
The present background is provided as illustrative environmental context only. It will be readily apparent to those of ordinary skill in the art that the principles and concepts of the present disclosure may be implemented in other environmental contexts equally, without limitation.
The present disclosure provides an in-vehicle system that alerts a driver to the presence and position of a detected emergency vehicle. The emergency vehicle is detected by the ego vehicle using both video and audio methodologies.
In one illustrative embodiment, the present disclosure provides a system for providing in-vehicle emergency vehicle detection and positional alerts, the system including: a camera coupled to an ego vehicle and configured to obtain an image of surroundings of the ego vehicle; an emergency vehicle recognition and localization module coupled to the camera and operable for segmenting an emergency vehicle from the image of the surroundings in order to detect and locate the emergency vehicle relative to the ego vehicle; a microphone coupled to the ego vehicle and configured to obtain an auditory signal from the surroundings of the ego vehicle; a siren detection and directional positioning module coupled to the microphone and operable for discriminating an emergency vehicle siren from the auditory signal from the surroundings in order to detect and locate the emergency vehicle relative to the ego vehicle; and one or more of a visual alert, an audible alert, and a haptic alert operable for alerting a driver of the ego vehicle to a presence and the location of the emergency vehicle relative to the ego vehicle responsive to output from the emergency vehicle recognition and localization module and the siren detection and directional positioning module.
In another illustrative embodiment, the present disclosure provides a method for providing in-vehicle emergency vehicle detection and positional alerts, the method including: obtaining an image of surroundings of an ego vehicle using a camera coupled to the ego vehicle; segmenting an emergency vehicle from the image of the surroundings using an emergency vehicle recognition and localization module coupled to the camera in order to detect and locate the emergency vehicle relative to the ego vehicle; obtaining an auditory signal from the surroundings of the ego vehicle using a microphone coupled to the ego vehicle; discriminating an emergency vehicle siren from the auditory signal from the surroundings using a siren detection and directional positioning module coupled to the microphone in order to detect and locate the emergency vehicle relative to the ego vehicle; and alerting a driver of the ego vehicle to a presence and the location of the emergency vehicle relative to the ego vehicle using one or more of a visual alert, an audible alert, and a haptic alert responsive to output from the emergency vehicle recognition and localization module and the siren detection and directional positioning module.
In a further illustrative embodiment, the present disclosure provides a non-transitory computer-readable medium stored in a memory and executed by a processor to carry out steps for providing in-vehicle emergency vehicle detection and positional alerts, the steps including: obtaining an image of surroundings of an ego vehicle using a camera coupled to the ego vehicle; segmenting an emergency vehicle from the image of the surroundings using an emergency vehicle recognition and localization module coupled to the camera in order to detect and locate the emergency vehicle relative to the ego vehicle; obtaining an auditory signal from the surroundings of the ego vehicle using a microphone coupled to the ego vehicle; discriminating an emergency vehicle siren from the auditory signal from the surroundings using a siren detection and directional positioning module coupled to the microphone in order to detect and locate the emergency vehicle relative to the ego vehicle; and alerting a driver of the ego vehicle to a presence and the location of the emergency vehicle relative to the ego vehicle using one or more of a visual alert, an audible alert, and a haptic alert responsive to output from the emergency vehicle recognition and localization module and the siren detection and directional positioning module.
The present disclosure is illustrated and described herein with reference to the various drawings, in which like reference numbers are used to denote like system components/method steps, as appropriate, and in which:
Again, the present disclosure provides an in-vehicle system that alerts a driver to the presence and position of a detected emergency vehicle. The emergency vehicle is detected by the ego vehicle using both video and audio methodologies.
Referring now specifically to
The image or images are provided to an emergency vehicle recognition and localization module 14 resident in a memory store of the ego vehicle 5 or in the cloud 7. The emergency vehicle recognition and localization module 14 implements a Convolutional Neural Network (ConvNet) and a combination of classical computer vision techniques, well known to those or ordinary skill in the art, to detect, localize, and track an emergency vehicle object in the image or images. Object detection may be used to identify objects within the images/video, typically outputting labels for multiple different items within the field of view of the camera system. For example, it is important for in-vehicle detection to be able to identify different types of vehicles, including cars, trucks, motorcycles, etc. Object tracking may be used to follow the particular object of interest, in this case one or more emergency vehicles, after initial object detection to give the driver of the ego vehicle 5 real time updates on the location of the emergency vehicle. In the case of a self-driving or driver-assisted ego vehicle 5, the system may also receive the location of the emergency vehicle and react accordingly. The emergency vehicle alert system 10 of the present disclosure may use these aforementioned computer vision techniques or a plurality of other methods known to one of ordinary skill in the art.
In order for a computer vision system to be operable, the system must be able to distinguish the object of interest. Emergency vehicles have prominent visual features, especially when the flashing lights are on. Thus, such emergency vehicles may be easily identified, located, and tracked relative to the ego vehicle 5. Multiple cameras 12 also allow for greater field of view and perception accuracy, as images may be combined, compared, and otherwise used synergistically.
The emergency vehicle alert system 10 of the ego vehicle 5 of the present disclosure also includes one or more microphones 16 that are configured to obtain an audio signal or signals from the surroundings of the ego vehicle 5. The one or more microphones 16 may include a front-facing microphone, a rear-facing microphone, a side-facing microphone, a directional microphone, a 360-degree microphone, and/or the like. The audio signal or signals are those typically used to detect the presence and position of a person or object outside the ego vehicle 5. The audio signal or signals are provided to a siren detection and directional positioning module 18 resident in the memory store of the ego vehicle 5 or in the cloud 7. The siren detection and directional positioning module 18 implements a Wavelength Neural Network (WaveNet) and classical computer hearing techniques, well known to those or ordinary skill in the art, to detect, localize, and track the emergency vehicle using the audio signal or signals. Emergency vehicles make distinct sounds, especially when the sirens are on. Thus, such emergency vehicles may be easily identified, located, and tracked relative to the ego vehicle 5. Multiple microphones 16 also allow for greater field of hearing and perception accuracy, as well as the use of triangulation techniques.
The visual recognition and audio recognition above are fused 20 to confirm the presence, location, and direction of travel of the emergency vehicle and an appropriate visual alert or alarm 22 and/or auditory alert or alarm 24 is/are issued to the driver in the ego vehicle 5. The visual alert or alarm 22 may consist of an appropriate display and/or warning light, and the auditory alert or alarm 24 may consist of an appropriate audible sound. A haptic alert or alarm may also be used in conjunction with the visual alert or alarm 22 and/or auditory alert or alarm 24. It should be noted that any alert or alarm utilized may be progressive, escalating from a gently “nudge” to an urgent “insistence” to an ego vehicle-initiated driver-assistance or self-driving operational intervention executed via the ego vehicle's advanced driver assistance system (ADAS) or autonomous driving (AD) and braking/steering systems. The fusion module and process 20 are operable for determining a degree of agreement between the visual recognition and the audio recognition, with significant thresholded disagreements being flagged. Further, the fusion module and process 20 can supplement one of the visual recognition and the audio recognition with the other, thereby enhancing the collective certainty and accuracy of the two.
It is to be recognized that, depending on the example, certain acts or events of any of the techniques described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.
The cloud-based system 100 can provide any functionality through services such as software-as-a-service (SaaS), platform-as-a-service, infrastructure-as-a-service, security-as-a-service, Virtual Network Functions (VNFs) in a Network Functions Virtualization (NFV) Infrastructure (NFVI), etc. to the locations 110, 120, and 130 and devices 140 and 150. Previously, the Information Technology (IT) deployment model included enterprise resources and applications stored within an enterprise network (i.e., physical devices), behind a firewall, accessible by employees on site or remote via Virtual Private Networks (VPNs), etc. The cloud-based system 100 is replacing the conventional deployment model. The cloud-based system 100 can be used to implement these services in the cloud without requiring the physical devices and management thereof by enterprise IT administrators.
Cloud computing systems and methods abstract away physical servers, storage, networking, etc., and instead offer these as on-demand and elastic resources. The National Institute of Standards and Technology (NIST) provides a concise and specific definition which states cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing differs from the classic client-server model by providing applications from a server that are executed and managed by a client's web browser or the like, with no installed client version of an application necessarily required. Centralization gives cloud service providers complete control over the versions of the browser-based and other applications provided to clients, which removes the need for version upgrades or license management on individual client computing devices. The phrase “software as a service” (SaaS) is sometimes used to describe application programs offered through cloud computing. A common shorthand for a provided cloud computing service (or even an aggregation of all existing cloud services) is “the cloud.” The cloud-based system 100 is illustrated herein as one example embodiment of a cloud-based system, and those of ordinary skill in the art will recognize the systems and methods described herein are not necessarily limited thereby.
The processor 202 is a hardware device for executing software instructions. The processor 202 may be any custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the server 200, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions. When the server 200 is in operation, the processor 202 is configured to execute software stored within the memory 210, to communicate data to and from the memory 210, and to generally control operations of the server 200 pursuant to the software instructions. The I/O interfaces 204 may be used to receive user input from and/or for providing system output to one or more devices or components.
The network interface 206 may be used to enable the server 200 to communicate on a network, such as the Internet 104 (
The memory 210 may include any of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, etc.), and combinations thereof. Moreover, the memory 210 may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that the memory 210 may have a distributed architecture, where various components are situated remotely from one another but can be accessed by the processor 202. The software in memory 210 may include one or more software programs, each of which includes an ordered listing of executable instructions for implementing logical functions. The software in the memory 210 includes a suitable operating system (O/S) 214 and one or more programs 216. The operating system 214 essentially controls the execution of other computer programs, such as the one or more programs 216, and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. The one or more programs 216 may be configured to implement the various processes, algorithms, methods, techniques, etc. described herein.
It will be appreciated that some embodiments described herein may include one or more generic or specialized processors (“one or more processors”) such as microprocessors; central processing units (CPUs); digital signal processors (DSPs); customized processors such as network processors (NPs) or network processing units (NPUs), graphics processing units (GPUs), or the like; field programmable gate arrays (FPGAs); and the like along with unique stored program instructions (including both software and firmware) for control thereof to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of the methods and/or systems described herein. Alternatively, some or all functions may be implemented by a state machine that has no stored program instructions, or in one or more application-specific integrated circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic or circuitry. Of course, a combination of the aforementioned approaches may be used. For some of the embodiments described herein, a corresponding device in hardware and optionally with software, firmware, and a combination thereof can be referred to as “circuitry configured or adapted to,” “logic configured or adapted to,” etc. perform a set of operations, steps, methods, processes, algorithms, functions, techniques, etc. on digital and/or analog signals as described herein for the various embodiments.
Moreover, some embodiments may include a non-transitory computer-readable storage medium having computer-readable code stored thereon for programming a computer, server, appliance, device, processor, circuit, etc. each of which may include a processor to perform functions as described and claimed herein. Examples of such computer-readable storage mediums include, but are not limited to, a hard disk, an optical storage device, a magnetic storage device, a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, and the like. When stored in the non-transitory computer-readable medium, software can include instructions executable by a processor or device (e.g., any type of programmable circuitry or logic) that, in response to such execution, cause a processor or the device to perform a set of operations, steps, methods, processes, algorithms, functions, techniques, etc. as described herein for the various embodiments.
The processor 302 is a hardware device for executing software instructions. The processor 302 can be any custom made or commercially available processor, a CPU, an auxiliary processor among several processors associated with the user device 300, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions. When the user device 300 is in operation, the processor 302 is configured to execute software stored within the memory 310, to communicate data to and from the memory 310, and to generally control operations of the user device 300 pursuant to the software instructions. In an embodiment, the processor 302 may include a mobile optimized processor such as optimized for power consumption and mobile applications. The I/O interfaces 304 can be used to receive user input from and/or for providing system output. User input can be provided via, for example, a keypad, a touch screen, a scroll ball, a scroll bar, buttons, a barcode scanner, and the like. System output can be provided via a display device such as a liquid crystal display (LCD), touch screen, and the like.
The radio 306 enables wireless communication to an external access device or network. Any number of suitable wireless data communication protocols, techniques, or methodologies can be supported by the radio 306, including any protocols for wireless communication. The data store 308 may be used to store data. The data store 308 may include any of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, and the like)), nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, and the like), and combinations thereof. Moreover, the data store 308 may incorporate electronic, magnetic, optical, and/or other types of storage media.
Again, the memory 310 may include any of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatile memory elements (e.g., ROM, hard drive, etc.), and combinations thereof. Moreover, the memory 310 may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that the memory 310 may have a distributed architecture, where various components are situated remotely from one another, but can be accessed by the processor 302. The software in memory 310 can include one or more software programs, each of which includes an ordered listing of executable instructions for implementing logical functions. In the example of
Referring now specifically to
Although the present disclosure is illustrated and described herein with reference to illustrative embodiments and specific examples thereof, it will be readily apparent to those of ordinary skill in the art that other illustrative embodiments and examples may perform similar functions and/or achieve like results. All such equivalent illustrative embodiments and examples are within the spirit and scope of the present disclosure, are contemplated thereby, and are intended to be covered by the following non-limiting claims for all purposes.
Number | Date | Country | |
---|---|---|---|
63065531 | Aug 2020 | US |