Patent Application
20230242161
This application relates to systems and methods for conveying vehicle autonomy states to a driver via visual and/or audio messaging. It also relates to systems and methods for enabling a driver to control and manage transitions into, out of, and between autonomy modes. The system can provide advanced notice of impending autonomy mode changes, and warn of system failures.
Researchers and vehicle manufacturers have been developing self-driving technologies for many years. Commercial trucking continues to be one of the areas where autonomous vehicles will eventually become widespread. Sensors and/or wireless connections (such as vehicle-to-vehicle radio communication) can be used by such autonomous systems to follow a route, and locate and avoid other vehicles and obstacles and follow the road. However, safety considerations require user interfaces for such vehicles that remain complex and difficult to use for many drivers.
In an example implementation, a vehicle such as a truck may be controlled by both a human driver and autonomy logic. The vehicle is therefore equipped with the usual throttle pedal, brake pedal and steering wheel for a human to operate. Electronic actuator devices are coupled to these human-controlled inputs. The electronic actuators in turn operate the vehicle's throttle, brake and steering sub-system mechanisms.
The vehicle also includes autonomy logic that may follow lanes, or a pre-defined plan or which generates a plan for controlling the vehicle such as the path it should follow. The autonomy logic uses inputs from sensors such as cameras, lidars, radars, and data available from other inputs such as radios and position sensors to devise the plan. The autonomy logic also provides an autonomy ready signal when it is able to transition into an autonomy mode based on its sensors and other inputs.
Autonomy control inputs are then generated to operate the throttle, brake, and steering sub-systems based on the plan. The controller also assesses the readiness of the actuators to accurately operate the vehicle's throttle, brake and steering sub-systems.
In some arrangements, a system controller receives both the human control inputs and the autonomy control inputs. These control inputs may be fed to a set of relays. The controller also receives the autonomy ready and actuator ready signals. The controller operates the relays to select which outputs to provide to an Electronic Control Unit (ECU) which in turn generates signals to be applied to the throttle, brake and steering actuators. The controller determines whether to set the relays to choose the human control inputs or autonomy control inputs based on several conditions such as the ready signals and/or other inputs such as human-operated mode input(s) and a human-operated stop input. In addition to switching between human and machine control, these relays can also indicate safety interlocks. For example, the control inputs may be fed to a set of electronic switches, relays, or equivalent devices to enable switching between manual and autonomous control. However, these control inputs may also be fed to a set of interlock devices to enable safe operation of the system.
The controller may also provide autonomy related information to the human driver via lights, tones, audio outputs and/or display screens associated with the autonomy system status, including state transitions into and out of autonomy modes. This information may include details such as whether the autonomy system is armed or disarmed, or in a pre-arm state and whether the autonomy logic is ready or not. The controller may also accept inputs from the driver to engage or disengage autonomy modes or enable only selected autonomy modes.
As shown more particularly in
The human driver 20 provides inputs to the system controller 70 via typical human input devices 40 such as a throttle pedal, brake pedal and steering wheel. It should be understood that the reference to a “throttle” herein does not mean the vehicle must be driven by an internal combustion engine. The vehicle may be propelled by electric motors or other alternatives to fossil fuels.
The human driver 20 can also view a display 50 and operate other inputs such as stop button 60 or mode input 75.
The autonomy logic 10 receives inputs from sensors 15 such as one or more camera(s), lidar(s), radar(s), position sensor(s), and/or receive data from other sources via radio(s) and other inputs. The autonomy logic 10 produces autonomy control signals including throttle, brake and steering signals. outputs to the controller 70.
The autonomy logic 20 may include an A-kit module 20 and a B-kit module 30.
The A-kit module 20 is responsible for generating instructions that describe a plan for the vehicle, such as a path that it should follow. The A-kit module also provides a ready signal when the autonomy logic 10 determines that it has sufficient information to devise and approve of a plan for the vehicle to be autonomously driven.
The B-kit module 30 receives inputs from the autonomy logic 10, such as instructions from the A-kit (including a path to follow) and produces autonomy control signals (for example including throttle, brake and steering control input signals) to the controller 70.
The controller 70 receives human control inputs 40 from the human driver 20 and autonomous control inputs from the B-kit module 30, and feeds at least the throttle to a Pulse Width Modulated (PWM) relay (not shown). The relay can select which of the inputs are fed to the vehicle's Electronic Control Unit (ECU) 80. Steering and brake inputs may be controlled over a Controller Area Network (CAN) bus.
The ECU 80 in turn produces electronic control signals used by one or more actuators 90 which may include the vehicles throttle 91, brake 92 and steering 93 actuators that in turn operate the physical throttle, brake and steering sub-systems on the vehicle. In some implementations the ECU may not control one or more of the actuators, such as the steering 93.
The controller 70 may also receive inputs from actuators 90 that indicate their current status.
In some implementations the controller 70 may provide visual or audio alerts outputs to the output device(s) 50. The output device(s) may include indicator lights and/or a speaker and electronics that can playback pre-recorded audio messages.
The controller 70 may also receive inputs such as from the stop button 60 and or mode switch 75 as operated by the driver 20.
The autonomy controller 70 may assume a number of states as follows:
Table 1 below lists the various controls and outputs for each of these states in more detail. The “Driver Alert System” referenced in Table 1 may include the output device 50 such as the electronics and speaker that play back pre-recorded audio messages. The “PWM controller” referenced in the table is portion of controller 70 that manages the autonomous control of the throttle and selection of the manual or autonomous throttle inputs.
It should be understood the table above is but one example. In other implementations, modes can be set to cause either the human inputs to be inactive, the human inputs add to the autonomous inputs, or the human inputs cause an exit from the autonomous control when activated.
Briefly, pressing the E-stop button 60 in any state causes the system to enter the E-stop 306 state. When E-stop is released, the system enters pre-disarm 301. The system remains in this state until the B-kit ready signal (referred to as “B-kit enable in
Conditions causing an exit from any of the states are also shown in
In another example, if at any time the B-kit ready or A-kit ready signal goes inactive, the disarm 302 is entered. Likewise, if the arm rocker is moved to the disarm position, then disarm 302 is entered.
Tables 2-1 and 2-2 provide more details list of the types of exceptions other events that may initiate transitions from one state to another.
The controller 70 may also accept inputs from a mode switch 75. The mode switch may be a three position switch corresponding to three system states: normal, steering only, and speed only. The mode switch 75 outputs may be fed to the controller 70 to control autonomy system modes as shown in Table 3.
Coordinating Autonomy State Transitions with Other Vehicles
It can now be understood how the above system and method can be used to improve how the driver may better control and remain informed of the state of the vehicle's autonomy logic. However in some instances the autonomy state of other vehicles may also be of importance. One such scenarios is where one or more drivers are responsible for two or more semi-trucks traveling together in a convoy. The multiple trucks, multiple human drivers and multiple autonomy logics should each be informed of what the other is doing.
Consider for example where the human interface should keep the driver informed of transitions into and out of autonomy mode. When one vehicle is about to transition into the autonomy state, the driver of another vehicle may want to be at least alerted to that fact. In other scenarios, where safety may be a concern, a lead driver may need to approve of any following vehicle's transition into autonomy. But cognitive or information overload should be avoided to the extent possible especially where human drivers are involved. Therefore, any interface to inform and/or obtain consent from a driver should be minimally distractive and easy to operate.
Each of the vehicle's respective controllers may communicate with the other vehicle and share information such as via a Vehicle to Vehicle (V2V) radio link. Information may also be exchanged via a shared world model (SWM) as explained in more detail below. Information received from the V2V link or the shared world model is then leveraged to provide alerts to and obtain consent from the human driver(s).
In a first state 401, vehicle V2 requests entry into the autonomy state. This request can be conveyed over the V2V link or via updates to the SWM. In state 402 vehicle V1 receives the request from V2 and presents it to the human driver of V1. The presentation may be by playing an audio clip (e.g., “Vehicle V2 requests to enter autonomy”) or by turning on a “Request for Autonomy” light.
In state 404 the human driver of V1 approves the autonomy request such as perhaps via the rocker switch of
In state 407 the controller in V2 then proceeds with the transition from the disarm state to the arm state to the ready for autonomy state (
In state 412 the human driver of V1 consents the transition by audio command or pressing a switch and this approval is communicated via the SWM or the V2V link to the controller in vehicle V2. In state 415, V2 completes the transition from the ready for autonomy state to the autonomy state. This state information again can be maintained in the SWM or updated through the V2V link. In state 416 vehicle V1 may present an autonomy mode alert to inform its driver that V2 is now operating in autonomy mode.
An emergency alert is presented to the driver of V1 in state 502.
In state 503 (which may occur before, after, or simultaneously with state 502) an alert is also presented to the driver of vehicle V2. E-stop mode is entered and confirmed in state 805. This e-stop status is also presented to vehicle V1 in state 506 who made turn display or play an alert to its driver. In states 806 and 808 both vehicles are human driven.
Example Implementation that Supports Collaborative Behaviors Via Shared World Model
As mentioned above, a Shared World Model is one way for the drivers and autonomy logic in multiple vehicles to engage in collaborative decision-making. A detailed example implementation of such an SWM was described in the pending patent application referenced on page 1. That description is repeated here for convenience of the reader.
1.1 Information Flows
An example vehicle 110 may include sensors 112, actuators 114, V2V radio transceivers 116, other I/O devices 118, and one or more processors 120 (also referred to herein as controllers 120). As discussed further herein, the one or more processors 120 may execute various logic including autonomy logic 122, decision logic 124 and may also maintain a world model 126. Vehicles 110 include human-driven vehicles 110-H and autonomous vehicles 110-A.
1.2. Collaboration Via Shared Information
As shown in
The controller 120 may implement algorithms that enable driver(s) and vehicles to collaborate based on the shared information. This information typically includes sensor data originating outside the components of the computer/human system as well as derived data (states, events, constraints, conclusions) originating inside the computer/human system and data indicative of physical phenomena created by human drivers or for the purpose of being sensed by human drivers. The shared information may therefore include data that (i) originates within or outside the convoy, (ii) represents physical phenomena (such phenomena produced by or capable of being sensed by humans, such as forces on steering wheels), (iii) is received from sensors or other input devices in its raw/sensed form or (iv) is derived data (examples include states, events, constraints, conclusions, originating inside the components of autonomy logic or human control).
Each vehicle 110 will have its own local copy 126 of such shared information referred to herein as a shared world model 240. At any given instant, each local copy 126 of the shared world model 240 may not be entirely consistent with the local copy 126 on other vehicles. Nevertheless, processes residing on all controllers 120 for all vehicles 110 attempt to keep the shared information in the shared world model 240 and local copies 126 sufficiently up-to-date and consistent to permit effective collaboration. Propagation of the local copy 126 of the shared world model 240 among vehicles 110 is discussed in more detail below.
1.3. Information Processing Steps
In
The perception and state estimation step 320 may process all of the information incoming from all sources in order to derive substantially new information that describes arbitrary attributes of the vehicles, humans, and external objects and traffic, etc. Such processing may comprise operations such as, for example:
The situation awareness and assessment step 322 may process all of the information incoming from all sources in order to derive substantially new information that is less directly related to sensed and communicated information, for example:
The decision making step 324 may process all of the information incoming from all sources in order to derive substantially new information that is associated with or comprises decision making, for example:
The behavior execution step 328 may process all of the information incoming from all sources in order to derive substantially new information that is associated with or causes acting in the real world, for example:
1.4. Shared World Model
In
The shared world model 240 comprises all information that is shared. In
As depicted in the example data flow diagram of
Collaborative Autonomy Transition Behavior Enabled by Shared Information
Given the context of shared information stored in a shared world model 240, it can now be understood how such shared information can be used to implement collaborative behaviors such as a convoy vehicle's transitions into or out of autonomy mode.
It should be understood that the example embodiments described above may be implemented in many different ways. In some instances, the various “data processors” may each be implemented by a physical or virtual general purpose computer having a central processor, memory, disk or other mass storage, communication interface(s), input/output (I/O) device(s), and other peripherals. The general-purpose computer is transformed into the processors and executes the processes described above, for example, by loading software instructions into the processor, and then causing execution of the instructions to carry out the functions described.
As is known in the art, such a computer may contain a system bus, where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. The bus or busses are essentially shared conduit(s) that connect different elements of the computer system (e.g., one or more central processing units, disks, various memories, input/output ports, network ports, etc.) that enables the transfer of information between the elements. One or more central processor units are attached to the system bus and provide for the execution of computer instructions. Also attached to system bus are typically I/O device interfaces for connecting the disks, memories, and various input and output devices. Network interface(s) allow connections to various other devices attached to a network. One or more memories provide volatile and/or non-volatile storage for computer software instructions and data used to implement an embodiment. Disks or other mass storage provides non-volatile storage for computer software instructions and data used to implement, for example, the various procedures described herein.
Embodiments may therefore typically be implemented in hardware, custom designed semiconductor logic, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), firmware, software, or any combination thereof.
In certain embodiments, the procedures, devices, and processes described herein are a computer program product, including a computer readable medium (e.g., a removable storage medium such as one or more DVD-ROM's, CD-ROM's, diskettes, tapes, etc.) that provides at least a portion of the software instructions for the system. Such a computer program product can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable, communication and/or wireless connection.
Embodiments may also be implemented as instructions stored on a non-transient machine-readable medium, which may be read and executed by one or more procedures. A non-transient machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a non-transient machine-readable medium may include read only memory (ROM); random access memory (RAM); storage including magnetic disk storage media; optical storage media; flash memory devices; and others.
Furthermore, firmware, software, routines, or instructions may be described herein as performing certain actions and/or functions. However, it should be appreciated that such descriptions contained herein are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.
It also should be understood that the block and system diagrams may include more or fewer elements, be arranged differently, or be represented differently. But it further should be understood that certain implementations may dictate the block and network diagrams and the number of block and network diagrams illustrating the execution of the embodiments be implemented in a particular way.
Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and/or some combination thereof, and thus the computer systems described herein are intended for purposes of illustration only and not as a limitation of the embodiments.
The above description has particularly shown and described example embodiments. However, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the legal scope of this patent as encompassed by the appended claims.
This patent application relates a co-pending U.S. patent application entitled “SHARED CONTROL FOR VEHICLES TRAVELLING IN FORMATION” Ser. No. 17/507,935 filed Oct. 22, 2021, which in turn claims the benefit of a provisional application entitled “SHARED CONTROL FOR VEHICLES TRAVELLING IN FORMATION” Ser. No. 63/128,961 filed Dec. 22, 2020, the entire contents of each of which are hereby incorporated by reference.
