MATERIAL HANDLING VEHICLE MONITORING SYSTEM AND METHOD

Abstract

A system and method for remotely monitoring a fleet of material handling vehicles is provided. The system and method include a remote monitoring station with a display interface and a server and one or more material handling vehicles with a sensor system and a vehicle controller with a processor and a memory device. The vehicle controller generates time-sequenced data based on received sensor data to create a composite image based on historical composite data and transmits the composite image in real-time to the display interface of the remote monitoring station.

Description

TECHNICAL FIELD

The present disclosure is related to the operation and monitoring of a fleet of material handling vehicles.

BACKGROUND

The use of robotic or autonomous vehicles in manufacturing and material handling applications is expanding rapidly as technology advances. The ability to adapt software to autonomous vehicles to improve decision making and perform more and more complex tasks is a driver of productivity. For example, having autonomous vehicles in a fleet cooperate and avoid dangerous interactions allows more autonomous vehicles to be used in a single workspace.

The ability for fleet members to interact and complete more complex tasks also presents additional opportunities for autonomous vehicles to encounter error conditions that they are neither programmed to recognize or resolve. Unexpected conditions, such as an unexpected impediment in a workspace or a malfunction of a particular subsystem of the autonomous vehicle can present problems that magnify within a fleet environment, such as for example, if a malfunctioning autonomous vehicle blocks the workflow of other autonomous vehicles in the fleet.

Failures of autonomous vehicle subsystems or error conditions can cripple a fleet operation in short order if not resolved. In many cases, resolution requires a human interaction that requires the human to enter the workspace and resolve the issue. The introduction of the human into the autonomous vehicle workspace presents new obstacles for the fleet members as well as potentially exposing the human to injury by one of the autonomous vehicles. This is confounded by the issue that many autonomous vehicles do not have readily available user controls to operate the autonomous vehicle and, even if present, the autonomous vehicle is not likely to be adapted for an operator to manually operate the autonomous vehicle.

Still further, the need to have humans on “stand-by” reduces the productivity gains provided by a fleet of autonomous vehicles. When operating properly, autonomous vehicle fleets need little to no intervention in the normal activities. For example, autonomous vehicles may be programmed to return to charging/fueling stations when needed and may take themselves out of service when preventative maintenance is required. Thus, there is no need for humans to be present when the fleet, or any particular autonomous vehicle, is operating correctly. There is a need to have humans available in the vicinity of the autonomous vehicles if error conditions arise.

Improved performance of autonomous vehicles and the reduction of the need for nearby humans to resolve errors and system malfunctions in autonomous vehicles would provide additional productivity and cost reductions, as well as reducing the potential for injury to humans. Still further, due to the limited number of issues that occur in a particular workspace, reducing or eliminating the need for human physical interaction presents the potential for additional productivity gains.

SUMMARY

The present disclosure includes one or more of the features recited in the appended claims and/or the following features which, alone or in any combination, may comprise patentable subject matter.

A material handling vehicle tracking and monitoring system is provided. The system includes a remote monitoring station and a material handling vehicle in communication with the remote monitoring station. The remote monitoring station includes a display interface and a server. The material handling vehicle includes one or more sensors with input parameters, a drive system that includes a drive and a drive controller, and a vehicle controller that includes a processor and a memory device. The vehicle controller includes programmable instructions executable by the processor that when executed cause the vehicle controller to perform various functions. The processor receives data from one or more sensors of the material handling vehicle and processes the data. The data is stored in the memory device of the vehicle controller. The processor generates time-sequenced data based on one or more sensors of the material handling unit and combines the time-sequenced data to provide a historical composite of data. A sequence of one or more composite images is created based on historical composite data. The composite image is transmitted to a display interface of the remote monitoring station.

In some embodiments, the system also includes a camera system designed to generate a signal representative of images of an environment surrounding the material handling vehicle. In some aspects, the processor receives signals from one or more sensors and from the camera system. The signals are aggregated and used to create a data array that is provided in a form of a synchronized composite of sensor signals and camera signals. The synchronized composite represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system. The time-sequenced composite image file is transmitted in real-time to the display interface of the remote monitoring station. In some forms, the remote monitoring station transmits a command signal to the material handling vehicle, and a camera associated with the material handling vehicle simultaneously changes the point of time at which the time-sequenced data is presented. In some embodiments, the remote monitoring station includes a headset, and a position of the headset is calibrated from data in an environment surrounding the material handling vehicle. In some aspects, the position of the headset is calibrated from a neutral position relative to a particular camera, a second material handling vehicle, or a combination thereof. In some forms, a user input device of the remote monitoring station includes an input for varying a point in time which corresponds to a portion of an image file associated with the composite image being transmitted, such that a user may choose to view the portion of the image file as it existed a different time from current real-time. In one or more embodiments, the portion of the image file transmitted is responsive to the position of the headset, such that the user may vary the field of view at different times to view the surroundings of the vehicle at that point in time. In one or more forms, the time sequence may be paused such that a user may look around the vehicle at a single point in time by moving the head set to change the field of view. In some embodiments, the material handling vehicle is designed to transmit an alert condition to the remote monitoring station associated with a time of the alert condition, such that a user can select the time of the alert condition to view images from a camera system or the data from the one or more sensors of the material handling vehicle.

A method for monitoring a material handling vehicle is provided. The method includes providing a remote monitoring station and a material handling vehicle. The remote monitoring station includes a display interface and a server. The material handling vehicle is in communication with the remote monitoring station. A processor of a vehicle controller of the material handling vehicle receives data from one or more sensors of the material handling vehicle. The processor of the vehicle controller processes and stores the data in a memory device of the material handling vehicle. The processor generates time-sequenced data to provide a historical composite of the data and creates a composite image based on the historical composite data. The composite image is transmitted to the display interface of the remote monitoring station.

In some embodiments, the remote monitoring station is monitoring updated real-time location and status indicators of the material handling vehicle. In some forms, the remote monitoring station transmits a color to the display, wherein the color is based on an error condition and urgency for the material handling vehicle. In some aspects, the remote monitoring station detects a critical event from one or more sensors on the material handling vehicle. In one or more embodiments, the remote monitoring station generates a point cloud from three-dimensional data obtained from one or more sensors. In some aspects, the remote monitoring station presents the point cloud in an image plane.

A material handling vehicle tracking and monitoring system is provided. The system includes a remote monitoring system and a materail hadnling vehicle in communication with the remote monitoring station. The remote monitoring station includes a display interface and a server. The material handling vehicle includes one or more sensors with input parameters and on-board inertial measurement units (IMUs). The material handling vehicle includes a drive system, a camera system, and a vehicle controller. The drive sytem includes a drive and a drive controller. The camera system includes one or more cameras. The vehicle controller includes a processor and a memory device. The vehicle controller includes programmable instructions executable by the processor that when executed cause the vehicle controller to perform various functions. The processor receives first signals from one or more sensors of the material handling vehicle. The processor receives second signals from the camera system and aggregates the first signals and the second signals received from the one or more sensors and the camera system. The processor creates visual data using the aggregated signals to generate a time-sequenced composite image file. The processor transmits the time-sequenced composite image file in real-time to the display interface of the remote monitoring station.

In some aspects, the material handling vehicle includes a communication circuitry to communicate with the remote monitoring station. In some forms, the communication circuitry includes a wireless communication protocol to share data in real-time between the material handling vehicle and the remote monitoring station. In one or more embodiments, one or more sensors includes an array of cameras, wherein an operator may toggle between views of one or more material handling vehicles and each camera of the array of cameras at a coordinated point in time using a headset worn by the operator.

Additional features, which alone or in combination with any other feature(s), such as those listed above and/or those listed in the claims, can comprise patentable subject matter and will become apparent to those skilled in the art upon consideration of the following detailed description of various embodiments exemplifying the best mode of carrying out the embodiments as presently perceived.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description particularly refers to the accompanying figures in which:

FIG. 1 is a diagrammatic floor plan of a material storage facility including autonomous vehicles used to move materials between storage locations, a number of cameras positioned throughout the facility, and a remote monitoring station;

FIG. 2 is a block diagram of the components of an illustrative autonomous vehicle of FIG. 1;

FIG. 3 is a flow chart showing an algorithm used by the autonomous vehicle of FIG. 1 to manage sensor data from various sensors of the autonomous vehicle;

FIG. 4 is a block diagram of the remote monitoring station of FIG. 1;

FIG. 5 is diagrammatic representation of an augmented reality interface for a user positioned in the remote monitoring station;

FIG. 6 is a perspective view of a portion of a working environment as viewed from the augmented reality interface, with the field of view of the augmented reality interface superimposed to show the limitations of the field of view of augmented reality interface at a particular moment in time;

FIG. 7 is a composite image of the field of view from an autonomous vehicle as viewed by the augmented reality interface including data elements superimposed on the image to provide the user of the augmented reality interface with additional information;

FIG. 8 a is diagrammatic floor plan similar to FIG. 1 showing an obstruction positioned in the path of an autonomous vehicle;

FIG. 9 is a composite image similar to FIG. 7 showing an obstruction in the path of the autonomous vehicle-the obstruction being outside of the field of view of the augmented reality interface;

FIG. 10 is the composite image of FIG. 9 with the field of view adjusted to show the obstruction;

FIG. 11 is an example of a monitor screen of the remote monitoring station in normal operating conditions;

FIG. 12 is an example of a monitor screen of the remote monitoring station in an alert condition that has been identified due to the detection of an impact;

FIG. 13 is a diagram of the data flow for creating the composite image of FIG. 7; and

FIG. 14 is a diagram of the communications between a particular autonomous vehicle and the remote monitoring station.

DETAILED DESCRIPTION

According to the present disclosure, a material storage facility 10 includes a number of storage racks 12, 14, 16, 18 positioned in a workspace 20 of the material storage facility 10. The material storage facility 10 also includes a number of autonomous vehicles 22 that operate within the material storage facility 10 to process materials in the material storage facility 10, including, in some embodiments, moving materials 24 from one storage location 26 to another storage location 28. It should be understood that the autonomous vehicles 22 may also move materials from the particular material storage facility 10 to another location outside of the material storage facility 10, such as to a manufacturing or distribution location outside of the material storage facility 10. In the embodiment of FIG. 1, the material storage facility 10 is illustrative only and the equipment and methods disclosed herein may be applied to the use of autonomous vehicles 22 in various facilities and applications.

The material storage facility 10 further includes a remote monitoring station (RMS) 30 positioned in the material storage facility 10. An operator 8 is located in the remote monitoring station 30, responsible for monitoring the fleet of autonomous vehicles 22 and resolving issues or error conditions. The positioning of the remote monitoring station 30 in the material storage facility 10 is for illustrative purposes only. The remote monitoring station 30 may be positioned outside of the material storage facility 10 at the same location, or may be positioned in a geographically distant location, such as a single remote monitoring station 30 facility that provides monitoring for various material storage facilities 10 around the world. The material storage facility 10 also includes a number of fixed cameras 32 positioned throughout the workspace 20. The number of fixed cameras 32 are positioned to provide viewing coverage of the workspace 20 in which the autonomous vehicles 22 operate.

Referring now to FIG. 2, the autonomous vehicle 22 includes a controller 34 that is responsible for operation of the particular autonomous vehicle 22. The controller 34 includes a processor 36 and memory 38. The processor 36 is in communication with the memory 38 and the memory 38 includes instructions that, when executed by the processor 36, cause the controller 34 to control the functionality of the autonomous vehicle 22. The processor 36 may be embodied as any type of processor capable of performing the functions described herein. For example, the processor 36 may be embodied as a single or multi-core processor(s), a single or multi-socket processor, a digital signal processor, a graphics processor, a microcontroller, or other processor or processing/controlling circuit. Similarly, the memory 38 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory 38 may store various data and software used during operation of the controller 34 such as operating systems, applications, programs, libraries, and drivers. The memory 38 is communicatively coupled to the processor 36 via the I/O subsystem 40, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor 36, the memory 38, and other components of the controller 34 or other components of the autonomous vehicle 22. For example, the I/O subsystem 40 may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem 40 may form a portion of a system-on-a-chip (SoC) and be incorporated, along with the processor 36, the memory 38, and other components of the controller 34 on a single integrated circuit chip.

Autonomous vehicle 22 includes a number of sensors 150 providing data to the controller 34. The sensors 150 include an array of cameras 42 positioned in various locations on the autonomous vehicle 22 to provide a three-hundred-sixty degree view of overlapping fields of view to provide an remote operator 8 full access to view the surroundings of the autonomous vehicle 22. The cameras 42 are embodied as RGBD cameras, providing enhanced information controller 34 that will be used as discussed below. In some embodiments, the autonomous vehicle may also include an array of RGB cameras 142 that provide visual data without the enhanced depth of the RGBD cameras 42. The autonomous vehicle 22 also includes one or more on-board inertial measurement units (IMUs) 44 to provide input to the controller 34 regarding changes in acceleration and orientation of the autonomous vehicle 22 in three-space during use. Still further, the autonomous vehicle 22 also includes an array of contact sensors 46 positioned on the autonomous vehicle 22 so that any physical contact between the autonomous vehicle 22 and some other item in the environment can be detected. Using the array of cameras 42, accelerometer(s) 44, and contact sensor array 46, the controller 34 is operable to sense the environment around the autonomous vehicle 22 to control the autonomous vehicle 22 as it completes its mission. In addition, the array of cameras 42, accelerometer(s) 44, and contact sensor array 46 allows the controller 34 to identify unexpected conditions that may require a response by the autonomous vehicle 22 or an intervention by an operator 8 positioned at the remote monitoring station 30. In some embodiments, the autonomous vehicle 22 may also include an array of LiDAR sensors 54 positioned about the autonomous vehicle 22 and operable to provide additional information to the controller 34 regarding the environment about the autonomous vehicle 22. Still further, in some embodiments, the autonomous vehicle 22 may also include an array of sonar sensors 56 positioned about the autonomous vehicle 22 and operable to provide additional information to the controller 34 regarding the environment about the autonomous vehicle 22. Additional sensors or sensor systems 150 of the autonomous vehicle 22 may include encoders 144 for measuring movement of components of the autonomous vehicle 22 or radar sensors 140. Other sensors 152 may be used as required by a particular application.

Also shown in the block diagram of FIG. 2, the autonomous vehicle 22 includes a drive system 49 which is operated, under the control of the controller 34 to move and steer the autonomous vehicle 22. The drive system 49 includes a drive 50 controlled by a drive controller 48 that communicates with controller 34 and provides the control signals for the drive 50. It should be understood that the autonomous vehicle 22 could have other functionality, in addition to the drive 50, including a manipulation system 136 which may embodied as various implements such a fork-lift mast system, articulators to move materials onto and off from the autonomous vehicle 22, or any of a number of other functions and sub-systems that adapt the autonomous vehicle 22 for specific purposes. The variation in functionality is limited only by the technical capabilities of a particular physical arrangement and the principles disclosed herein may be used with any fleet of autonomous vehicles 22 that interact within a pre-defined workspace, such as workspace 20, for example.

The autonomous vehicle 22 also includes communication circuitry 52 that allows the autonomous vehicle 22 to interact with the remote monitoring station 30 and to share data there between. The communications circuitry 52 can take many forms and be similar to the input/output controller 40 or be modified for a particular application. It is contemplated that the communications circuitry 52 would include a high-speed wireless communications protocol to allow data to be shared between the controller 34 and the remote monitoring station 30 in real-time to allow the autonomous vehicle 22 to operate at relatively high speeds in the workspace 20.

As will be explained in further detail below, the array of cameras 42, accelerometer(s) 44, and contact sensor array 46, and optionally the LiDAR array 54 and/or sonar array 56 provide signals to the controller 34 that are then processed and stored in memory 38 to provide a time sequenced data based on the inputs from the various sensors 42, 44, 46, 54, and 56 that can be combined to provide a historical composite of the data that can create a composite image and status that allows a user to reconstruct the environment of the autonomous vehicle 22 at an earlier point in time. This is useful for an operator 8 to diagnose a particular problem experienced by an autonomous vehicle 22 by viewing the surroundings of the autonomous vehicle 22 at an earlier point in time. The data can also be combined to create a composite image available to the operator 8 at the remote monitoring station 30 in real-time so that the operator 8 may manually operate the autonomous vehicle 22 using the combined data.

The autonomous vehicle 22 also includes an independent safety system 138 with safety rated sensors that are operable to limit operation of the autonomous vehicle 22. The autonomous vehicle 22 includes a battery 88 which powers the various components of the autonomous vehicle 22. The battery 88 interfaces with a battery relay 134 which is operable to interrupt power to both the manipulation system 136 and drive system 49 to prevent operation of those systems in critical safety situations. If the safety system 138 detects an unsafe condition, the power to the manipulation system 136 and drive system 49 is cut until the unsafe condition is resolved.

The remote monitoring station 30 is shown diagrammatically in FIG. 4 and includes a workstation 56 that includes a computer 152, a display 58, a QWERTY keyboard 60, and a mouse 62. The display 58 may include multiple discrete display devices or may be segmented to provide views of status screens of multiple material storage facilities 10 simultaneously. For example, referring to FIG. 8, a status screen of a single material storage facility 10 is shown. The display 58 may be configured to simultaneously show each material storage facility 10 the operator 8 is monitoring updated with real-time location and status indicators of the various autonomous vehicles 22. For example, each autonomous vehicle 22 that is not in an error condition could be shown in one color, such as green, for example. Any autonomous vehicles 22 that is experiencing an error condition, such as the specific autonomous vehicle 22′ shown in FIG. 8, may be shown in a different color, such as yellow or red, for example, depending on the urgency of the error condition. In some embodiments, the display 58 is configured as a touchscreen display, permitting an operator 8 to interact directly with the display to provide inputs to the workstation 58. The workstation 58 may be a special purpose computer device, or may be a general purpose computer device with operating systems, firmware, and software specially configured to perform the functions described herein.

The remote monitoring station 30 also includes an augmented reality interface 64 that is worn by an operator 8 when the operator 8 is monitoring the one or more material storage facilities 10 as shown in FIG. 5. The augmented reality interface 64 includes a head set 66, and, optionally, may include a pair of gesture sensors 68, 70 worn by the operator 8 or position tracking sensor 146 that monitors movements and gestures of the operator 8. The head set 66 provides a heads up display of information presented to a display 72 of the head set 66. In operation, the augmented reality interface 64 is not active during normal operations of a group of autonomous vehicles 22 being monitored by an operator 8. The operator 8 is able to see through the display 72 and perform normal procedures in the remote monitoring station 30. However, if a particular autonomous vehicle 22 experiences an error condition, the operator 8 may select the particular autonomous vehicle 22 on the display 58 of the workstation 56 to activate the augmented reality interface 64 as a virtual controller for the particular autonomous vehicle 22. When the autonomous vehicle 22 is selected, the augmented reality interface 64 activates the display 72 and places in the operator 8 in a virtual operator compartment for the particular autonomous vehicle 22, presenting a real-time augmented display using information from the array of cameras 42, accelerometer(s) 44, and contact sensor array 46, and optionally the LiDAR array 54 and/or sonar array 56 so that the operator 8 may operate the autonomous vehicle 22 virtually, from the remote monitoring station 30. As will be explained below, the view presented to the operator 8 will be dependent on the position and orientation of the operator's head so that as the operator 8 moves her head in real space, the augmented view presented to the operator 8 on the display 72 changes with the signals from the array of cameras 42, accelerometer(s) 44, and contact sensor array 46, and optionally the LiDAR array 54 and/or sonar array 56 being presented in a composite image that allows the operator 8 to comprehend the current state of the environment, including the relative distance of certain elements of the environment surrounding the autonomous vehicle 22. The gesture sensors 68, 70 allow the operator 8 to activate virtual controls for the autonomous vehicle 22, such as a virtual joystick driver controller, for example.

Referring now to FIG. 3, a process 160 for managing data collected by the sensors 150 of the autonomous vehicle 22 includes a start-up at step 162. Once the autonomous vehicle 22 is started at step 162, a start-up process step 164 initializes all of the functions and systems of the autonomous vehicle 22 are initialized. The process 160 then advances to a process step 166 where a pre-defined alerts configuration is applied to the controller 34. The alerts configuration is defined separately at a process 168 and transmitted to the controller 34 by the server 78 of the remote monitoring station 30. The pre-defined alerts configuration is established based on the operating environment of the particular autonomous vehicle 22.

Once the alerts configuration is loaded at step 166, the process 160 advances to a normal operating process 170. The process continuously progresses through a decision tree where the data from the sensors 150 is evaluated to determine if a critical event has occurred at decision step 172. If no critical event is detected, the process 160 progresses to a decision step 174 to determine if any important events have occurred. If a critical event is detected at decision step 172, then the process 160 advances to step 176 wherein critical event data is saved to the server 78. Then the process 160 progresses to step 178 where the event data is displayed to the operator 8 at the remote monitoring station 30.

If no critical event is detected at step 172 and no important event is detected at step 174, then the process advances to step 180 where sensor data is temporarily saved as short-term data. This short-term data is not maintained beyond a predetermined period as the data set becomes overwhelming for the controller 34. However, the short-term event data is available to the operator 8 in certain situations as discussed below. If important data is detected at step 174, then the process 160 advances to step 178 discussed above and then advances to the step 180.

From step 180, the process advances to step 182 where the autonomous vehicle 182 operates based on the detected data. The operation is evaluated at decision step 184 to determine if operation of the autonomous vehicle 22 should continue. If the data indicates that the autonomous vehicle 22 is safe to operate, then the process 160 returns to step 170 and progresses through the decision tree described above. If the autonomous vehicle 22 is not safe to operate, then the decision step 184 initiates a shut-down and the process progresses to an end step 186 where the autonomous vehicle 22 is inoperable until the shut-down condition is resolved by the operator 8.

Referring to FIG. 14, a diagrammatic representation of the data flow between the autonomous vehicle 22, the server 78, and the workstation 56 is shown. The data from the sensors 150 is provided to the controller 34 of the autonomous vehicle 22 which shares that data as live event data 246 with the remote monitoring station 30. The data from the sensors 150 is also applied to an important message filter 230 imbedded in the software of the controller 34 and, when an important event is detected, the important event data 248 is transferred to server storage 236. If a critical event is detected by the critical event filter 232 of the controller 34, then the event data 248 is transferred to the server storage 236. In addition, upon detection of a critical event by the critical event filter 232, data stream 250 moves the data from the sensors 150 to a recording storage location 238 in the memory of the server 78. The event data 248 is applied to emergency filters 234 on the server 78 to determine if an emergency condition is detected. If so, alerts 242 are forwarded to the remote monitoring station 30. The data stream 250 placed in recording storage 238 is available to the remote monitoring station 30 through a transfer 244 of recorded event data. In addition, the remote monitoring station 30 is operable to update filter settings 240 which may be applied to the filters 230, 232 of the autonomous vehicle 22 by the server 78 at process step 166 discussed above.

Thus, when an event occurs requiring the operator 8 to assess the environment of the autonomous vehicle 22, the operator 8 may also be able to move back in time, virtually, to view the environment of the autonomous vehicle 22 at the carlier time based on the data saved to the autonomous vehicle at step 180 or the server at step 176. The operator 8 may move their head to view different fields of view around the autonomous vehicle 22 at that point in time, with the information on the display being presented as an augmented reality image. This is useful for the operator 8 to reconstruct the environment of the autonomous vehicle 22 prior to the error condition. The data from the array of cameras 42, accelerometer(s) 44, and contact sensor array 46, and optionally the LiDAR array 54 and/or sonar array 56 is stored in memory 38 and is used to reconstruct the image on the augmented reality interface 64 as if it were in real-time. As such, the operator 8 is able to detect any abnormalities, improving the ability to resolve the alert condition.

The workstation 56, through communications circuitry 76, is also in communication with the cameras 32 positioned throughout the workspace 20 and selectively accessible by the operator 8 to choose a particular camera 32 to view the workspace 20 from the perspective of the particular camera 32. A server 78 facilitates the communication between the workstation 56, cameras 32, and autonomous vehicles 22 so that all of the information is available to operator 8. This functionality is available in real-time such that when an error condition is triggered, such as with the autonomous vehicle 22′ in FIG. 8, for example, the operator 8 may choose a particular camera 32, such as camera 32′ in FIG. 8, that has the autonomous vehicle 22′ in its field of view to review the environment of the autonomous vehicle 22′ from the perspective of the camera 32. This may occur prior to the operator 8 engaging the augmented reality interface 64 for the autonomous vehicle 22′. In this way, the operator 8 will have some context as to the environment of the autonomous vehicle 22′ before becoming the virtual operator 8 of the autonomous vehicle 22′.

The data from the cameras 32 are maintained in memory 74 on the workstation 56 or at the server 78. When the operator 8 has time-shifted the information presented on the augmented reality interface 64 as discussed above, the workstation 56 receives the time-shift information and resets the images from the various cameras 32 to the same point in time as the time-shifted such that the operator 8 may toggle between the virtual operator condition and viewing the workstation 56 at the time-shift to thereby review the view from any of the cameras 32 to further diagnose the error condition. This further assists the operator 8 in diagnosing the error condition of the autonomous vehicle 22′.

Importantly, the workstation 56, through the server 78, is also operable to reset available information in all of the adjacent autonomous vehicles 22 to the time-shifted point in time. Thus, while the operator 8 is engaged with a virtual time-shift, any of the actions the operator 8 takes will be in reference to the particular point in time that the operator 8 has shifted to, until the time-shift is released. For example, the operator 8 can move from being the virtual operator 8 of the autonomous vehicle 22′ if FIG. 8 to the virtual operator 8 of the autonomous vehicle 22″ in FIG. 8 and will have the perspective of autonomous vehicle 22″ to view the autonomous vehicle 22′ as the operator 8 diagnoses an error condition. In this way, the available views and information for the operator 8 is expanded to include other members of the fleet of the autonomous vehicles 22. The time-shift may be completed by selecting a particular point in real-time, through typical A/V playback controls on the augmented reality interface 64 or workstation 56, or by an error message 126 as shown in FIG. 12. Because the error message is time-stamped, selecting the time-stamp will may automatically move the various information stored in memory 38 and/or memory 74 to that particular point in time to allow the operator 8 to quickly make the virtual time-shift in the augmented reality interface 64.

As shown in FIG. 6, the field of view 80 available on the display 72 is limited. It should be understood that the representations of the field of view 80 in FIGS. 6, 7, 9, and 10 that follow are presented as examples only. The shape and size of the field of view 80 may vary depending on application and the information presented in the augmented view presented on display 72 may vary depending on the needed information. In the view shown in FIG. 6, an arrow indicator 82 provides the operator 8 with a reference of the view being presented to the front of the particular autonomous vehicle 22. As shown in FIG. 10, the orientation of the arrow 82 changes as the operator 8 moves her head and changes the field of view. FIG. 6 shows a typical view of the environment around the autonomous vehicle 22. In practice, this display will normally be in high definition color. The view of FIG. 6 shows the autonomous vehicle 22″ from the view of autonomous vehicle 22′. It should be understood that the view of FIG. 6 omits background that would be visible to improve the clarity of the discussion here.

FIG. 7 shows the view of FIG. 6, with the display 72 being updated to include augmentation from the array of cameras 42, accelerometer(s) 44, and contact sensor array 46, and optionally the LiDAR array 54 and/or sonar array 56. In addition, other feedback may be presented such as ground speed 86 from the drive controller 48 or the level of charge 90 of the battery 88. The augmentation presented also includes indications of the distance of various objects and points as indicated by the dashed lines presented in FIG. 7. For example, the heavier dashed lines 92 in FIG. 7 indicate that the autonomous vehicle 22″ is relatively close to the autonomous vehicle 22′. The thinner lines 94 indicate that those features are farther away. Similarly, the dashed lines 96 indicate that the pallet 98 is relatively far away. In the present disclosure, the thickness of the lines 92, 94, and 96 varies to provide information regarding the relative distance. It should be understood that in implementation, the lines may be presented differently, including different colors and may also blink or flash at varying rates to provide appropriate feedback to the operator 8.

Referring to FIG. 12, an example of a status screen 110 shown on the display 58 of the workstation 56 would be communicated to the operator 8 with the autonomous vehicle 22′ being shown in a critical condition. The display includes a diagrammatic representation 112 of the material storage facility 10, which is designated as Plant 1. A status board 114 provides information about the various autonomous vehicles 22 in disparate material storage facilities 10, including Plant 1, a Plant 2, and a Plant 3. The first column 116 shows the number of autonomous vehicles 22 operating normally. The second column 118 shows the number of autonomous vehicles 22 in cach material storage facility 10 that has a cautionary condition. The column 120 shows the number of autonomous vehicles 22 with error conditions. A message screen 122 provides a textual explanation 126 of the error condition, along with a time-stamp 124.

FIG. 11 is similar to FIG. 12, but shows that there are no error conditions. However, as shown in FIG. 11, an autonomous vehicle 22W is shown is highlighted to indicate a warning condition. The message screen 122 shows a message 130 with a time-stamp 132 that indicates that the autonomous vehicle 22W is having battery performance issues.

FIG. 8 shows that the error condition of FIG. 12 is caused by an obstruction/box 100 that has fallen in the path of the autonomous vehicle 22′. However, because the obstruction 100 is not shown in the field of view of the augmented display 72 as shown in FIG. 9, the operator may move the field of view by moving her head as shown in FIG. 10 to see the obstruction 100. At this point, the operator 8 will be able to use the functionality described above to pick different views to see the obstruction 100 from either camera 32′ or from the perspective of autonomous vehicle 22″ and make efforts to resolve the issue.

FIG. 13 illustrates the process 210 of development of the composite images seen by the operator 8 when using the augmented reality interface 64. The process 210 includes the use of data 200 from the autonomous vehicle 22 or data 202 stored on the server 78 which is chosen at the “or” step 204. Data 206 regarding the performance of the autonomous vehicle 22 is transferred to a data receiver 212 of the server 78. In addition, data 208 from various sensors 150 presented in the composite image is also provided to the data receiver 212. The three-dimensional data is used to generate a point cloud 216 of three-dimensional data within the operator's field of view 218. The point cloud 216 is generated to establish an image plane 222 that presents the three-dimensional data in a two-dimensions at the image plane 222. Additional data is provided through a three-dimensional data stream analysis 214 of the server 78 so that additional data 220 is presented at the image plane 222 in composite with the point cloud 216 so that a composite image including image data and representation of the data from the sensors 150 is presented at the image plane 222, providing the operator 8 the composite image discussed above.

Although this disclosure refers to specific embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the subject matter set forth in the accompanying claims.

Claims

1. A material handling vehicle tracking and monitoring system comprising: a remote monitoring station including a display interface and a server; a material handling vehicle in communication with the remote monitoring station, the material handling vehicle comprising:one or more sensors with input parameters;a drive system including a drive and a drive controller, wherein the drive receives control signals from the drive controller;a vehicle controller with a processor and a memory device with programmable instructions executable by the processor that when executed cause the vehicle controller to: receive data from the one or more sensors of the material handling unit;process the data and store the data in the memory device of the vehicle controller;generate time-sequenced data based on the input parameters of the one or more sensors of the material handling unit;combine the time-sequenced data to provide a historical composite of the data;create a composite image based on the historical composite data; andtransmit the composite image to the display interface of the remote monitoring station.

2. The material handling vehicle tracking and monitoring system of claim 1, further comprising: a camera system designed to generate a signal representative of images of an environment surrounding the material handling vehicle.

3. The material handling vehicle tracking and monitoring system of claim 2, wherein the memory devices include instructions that, when executed by the processor, cause the processor to: receive sensor signals from the one or more sensors of the material handling vehicle;receive camera signals from the camera system;aggregate the sensor signals and the camera signals;create a data array that is provided in a form of a synchronized composite of the sensor signals and the camera signals, which represents a time-sequenced composite image file that superimposes data derived from the sensor signals onto the images from the camera system; andtransmit the time-sequenced composite image file in real-time to the display interface of the remote monitoring station.

4. The material handling vehicle tracking and monitoring system of claim 1, wherein the remote monitoring station transmits a command signal to the material handling vehicle and a camera associated with the material handling vehicle to simultaneously change a point of time that the time-sequenced data is presented.

5. The material handling vehicle tracking and monitoring system of claim 1, wherein the remote monitoring station includes a headset, and a position of the headset is calibrated from data in an environment surrounding the material handling vehicle.

6. The material handling vehicle tracking and monitoring system of claim 5, wherein the position of the headset is calibrated from a neutral position relative to a particular camera, a second material handling vehicle, or a combination thereof.

7. The material handling vehicle tracking and monitoring system of claim 1, wherein a user input device of the remote monitoring station includes an input for varying a point in time which corresponds to a portion of an image file associated with the composite image being transmitted, such that a user may choose to view the portion of the image file as it existed a different time from current real-time.

8. The material handling vehicle tracking and monitoring system of claim 1, wherein the portion of the image file transmitted is responsive to the position of the headset, such that the user may vary the field of view at the different time to view the surroundings of the vehicle at that point in time.

9. The material handling vehicle tracking and monitoring system of claim 1, wherein the time sequence may be paused such that a user may look around the vehicle at a single point in time by moving the head set to change the field of view.

10. The material handling vehicle tracking and monitoring system of claim 1, wherein the material handling vehicle is designed to transmit an alert condition to the remote monitoring station associated with a time of the alert condition, such that a user can select the time of the alert condition to view images from a camera system or the data from the one or more sensors of the material handling vehicle.

11. A method for monitoring material handling vehicles, comprising: providing a remote monitoring station including a display interface and a server;providing a material handling vehicle in communication with the remote monitoring station;receiving data from one or more sensors on the material handling vehicle via a processor of a vehicle controller of the material handling vehicle; processing the data using the processor;storing the data in the memory device of the vehicle controller;generating time-sequencing data to provide a historical composite of the data;creating a composite image based on the historical composite data; andtransmitting the composite image to the display interface of the remote monitoring station.

12. The method for monitoring material handling vehicles according to claim 11, further comprising: monitoring updated real-time location and status indicators of the material handling vehicle using the remote monitoring station.

13. The method for monitoring material handling vehicles according to claim 11, further comprising: transmitting to the display a color based on an error condition and urgency for the material handling vehicle using the remote monitoring station.

14. The method for monitoring material handling vehicles according to claim 11, further comprising: detecting a critical event from one or more sensors on the material handling vehicle using the remote monitoring station.

15. The method for monitoring material handling vehicles according to claim 11, further comprising: generating a point cloud from three-dimensional data obtained from one or more sensors using the remote monitoring station.

16. The method for monitoring material handling vehicles according to claim 15, further comprising: presenting the point cloud in an image plane using the remote monitoring station.

17. A material handling vehicle tracking and monitoring system comprising: a remote monitoring system including a display interface and a server;a material handling vehicle in communication with the remote monitoring station, the material handling vehicle comprising: one or more sensors with input parameters and on-board inertial measurement units;a drive system including a drive and a drive controller, wherein the drive receives control signals from the drive controller;a camera system including one or more cameras;a vehicle controller with a processor and a memory device with programmable instructions executable by the processor that when executed cause the vehicle controller to: receive first signals from the one or more sensors of the material handling vehicle;receive second signals from the camera system;aggregate the first signals and the second signals received from the one or more sensors and the camera system;create visual data using the aggregated signals to generate a time-sequenced composite image file; andtransmit a time-sequenced composite image file in real-time to the display interface of the remote monitoring station.

18. The material handling vehicle tracking and monitoring system of claim 17, wherein the material handling vehicle includes a communication circuitry to communicate with the remote monitoring station.

19. The material handling vehicle tracking and monitoring system of claim 18, wherein the communication circuitry includes a wireless communication protocol to share data between the material handling vehicle and the remote monitoring station in real-time.

20. The material handling vehicle tracking and monitoring system of claim 17, wherein the one or more sensors include an array of cameras, wherein an operator may toggle between views of one or more material handling vehicles and each camera of the array of cameras at a coordinated point in time using a headset worn by the operator.