Patent Application
20240192689
The subject matter described herein relates to systems and methods for autonomously controlling movement of a device.
The challenges in the modern vehicle yards are vast and diverse. Classification yards, or hump yards, play an important role as consolidation nodes in vehicle freight networks. At classification yards, inbound vehicle systems (e.g., trains) are disassembled and the cargo-carrying vehicles (e.g., railcars) are sorted by next common destination (or block). The efficiency of the yards in part drives the efficiency of the entire transportation network.
The hump yard is generally divided into three main areas: the receiving yard, where inbound vehicle systems arrive and are prepared for sorting; the class yard, where cargo-carrying vehicles in the vehicle systems are sorted into blocks; and the departure yard, where blocks of vehicles are assembled into outbound vehicle systems, inspected, and then depart.
Current solutions for field service operations are labor-intensive, dangerous, and limited by the operational capabilities of humans being able to make critical decisions in the presence of incomplete or incorrect information. Furthermore, efficient system level-operations require integrated system wide solutions, more than just point solutions to key challenges. The nature of these missions dictates that the tasks and environments cannot always be fully anticipated or specified at the design time, yet an autonomous solution may need the essential capabilities and tools to carry out the mission even if it encounters situations that were not expected.
Solutions for typical vehicle yard problems, such as brake bleeding, brake line lacing, coupling cars, etc., can require combining mobility, perception, and manipulation toward a tightly integrated autonomous solution. When placing robots in an outdoor environment, technical challenges largely increase, but field robotic application benefits both technically and economically.
One challenge in using automated robotic systems to perform maintenance of the vehicles in the yard is ensuring that the robotic systems safely move through the yard. For example, safeguards are needed to ensure that the robotic systems do not collide with other objects (stationary or moving) and that the robotic systems are able to respond to a dynamically changing environment (e.g., where an object moves into the path of a moving robotic system), while also attempting to ensure that the robotic systems move toward locations for performing the vehicle maintenance along efficient paths (e.g., the shortest possible path or the path that is shorter than one or more other paths, but not all paths)
In an embodiment, a system is provided that includes a robotic vehicle having a propulsion system configured to propel the robotic vehicle through an external environment and an actuator configured to perform designated operations while in the external environment. The system also includes one or more sensors disposed onboard the robotic vehicle configured to obtain environmental data representative of the external environment. The system also includes a local controller disposed onboard the robotic vehicle and configured to receive input signals from an off-board controller. Responsive to receiving an input signal from the off-board controller for moving in an autonomous mode, the local controller is configured to autonomously move the robotic vehicle within the external environment. Responsive to receiving an input signal for operating in a tele-operation mode, the local controller is configured to exit the autonomous mode, wherein the input signal for operating in the tele-operation mode includes a remote command that dictates at least one of a movement of the robotic vehicle or a movement of the actuator.
In an embodiment, a method is provided that includes obtaining, at a local controller disposed onboard a robotic vehicle, environmental data representative of an external environment of the robotic vehicle. The external environment includes a designated region where the robotic vehicle moves and performs operations. Responsive to receiving an input signal for moving the robotic vehicle in an autonomous mode, the method also includes autonomously generating control signals, by the local controller, for moving the robotic vehicle through the external environment. Responsive to receiving an input signal for operating in a tele-operation mode, the method further comprises exiting the autonomous mode, wherein the input signal for operating in the tele-operation mode includes a remote command that dictates at least one of a movement of the robotic vehicle or a movement of the actuator.
The present inventive subject matter will be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:
One or more embodiments of the inventive subject matter described herein provide robotic systems and methods that provide a large form factor mobile robot with one or more actuators. The actuators may effectively detect, identify, and subsequently engage or monitor components or sub-systems of a power system. The robotic systems may be used to inspect, maintain, and/or repair the power systems.
In one or more examples, the power systems may refer to stationary power systems or non-stationary power systems. The stationary power systems may include power generating systems such as wind turbines, or the like. The non-stationary power systems may include vehicle systems. While one or more embodiments are described in connection with a rail vehicle system, not all embodiments are limited to rail vehicle systems. Unless expressly disclaimed or stated otherwise, the inventive subject matter described herein extends to multiple types of vehicle systems. These vehicle types may include automobiles, trucks (with or without trailers), buses, marine vessels, aircraft, mining vehicles, agricultural vehicles, or other off-highway vehicles. The vehicle systems described herein (rail vehicle systems or other vehicle systems that do not travel on rails or tracks) can be formed from a single vehicle or multiple vehicles. With respect to multi-vehicle systems, the vehicles can be mechanically coupled with each other (e.g., by couplers) or logically coupled but not mechanically coupled. For example, vehicles may be logically but not mechanically coupled when the separate vehicles communicate with each other to coordinate movements of the vehicles with each other so that the vehicles travel together as a group. Vehicle groups may be referred to as a convoy, consist, swarm, flect, platoon, and train.
As used herein, the term “actuator” includes any mechanism by means of which something of the robotic system is moved. An actuator may be power or driven electrically, hydraulically, or pneumatically. As one example, an actuator may include an arm that is controlled to engage and disengage a component of a power system. As another example, the actuator may move a data collector of the robotic system to better position the data collector relative to a target (or object of interest). For instance, the data collector may include an image detector in which the target is a coupler between two vehicles. The target could also be a designated region of a power system in which the image data is used to determine if the power system needs maintenance or repair. The data collector can include an optical scanner in which the target is machine-readable data (e.g., bar codes) associated with the power system. The data collector may include a thermometer and the target may be fluid or a surface. The data collector may include an antenna and the target may be a near field communication (NFC) tag or radio frequency identification (RFID) tag. The actuator may include one or more motors that moves the data collector in a better position for obtaining the data. For example, the actuator may move the data collector closer to a target. The actuator may re-orient the data collector relative to a target.
As another example, an actuator may include a steering system (e.g., rack and pinion system and/or differential steering system) that turns the robotic system autonomously or tele-operationally in response to input signals from an operator.
As another example, the actuator may include a coupling device that engages the power system. For example, the coupling device may mechanically engage the power system by mating with designated features (e.g., lock and key). During or after mechanically engaging the power system, the coupling device may connect to at least one of pneumatic lines, electrical lines, or data transmission lines of the power system. For example, the coupling device may include a plug body having at least one of a pneumatic port, electrical port, or data transmission port. The actuator may move the plug body and insert the plug body into a receptacle of the power system where the receptacle has at least one of a pneumatic port, electrical port, or data transmission port. Alternatively, the actuator may move a receptacle that receives a plug body of the power system.
In some embodiments, the actuator may be used to perform one or more maintenance operations on vehicles, such as obtaining information from vehicles (e.g., AEI tag reading), inspecting vehicles (e.g., inspecting couplers between vehicles), air hose lacing, etc.
In one or more examples, the robotic systems can be controlled to inspect and/or repair the power systems, such as vehicles. While the descriptions herein focuses on manipulating brake levers of vehicles (e.g., rail vehicles) in order to bleed air brakes of the vehicles, not all maintenance operations performed by the robotic systems or using the methods described herein are limited to brake bleeding. One or more embodiments of the robotic systems and methods described herein can be used to perform other maintenance operations on vehicles, such as obtaining information from vehicles (e.g., AEI tag reading), inspecting vehicles (e.g., inspecting couplers between vehicles), air hose lacing, etc.
The robotic system 100 also includes several sensors 108, 109, 110, 111, 112 that measure or detect various conditions used by the robotic system 100 to move toward, grasp, and actuate brake levers. The sensors 108-111 are optical sensors, such as cameras, infrared projectors and/or detectors. While four optical sensors 108, 110 are shown, alternatively, the robotic system 100 may have a single optical sensor, less than four optical sensors, or more than four optical sensors. In one embodiment, the sensors 109, 111 are RGB cameras and the sensors 110, 112 are structured-light three-dimensional (3-D) cameras, but alternatively may be another type of camera.
The sensor 112 is a touch sensor that senses operation of an actuator, such as detecting when a manipulator arm 114 of the robotic system 100 contacts or otherwise engages a surface or object. The touch sensor 112 may be one or more of a variety of touch-sensitive devices, such as a switch (e.g., that is closed upon touch or contact), a capacitive element (e.g., that is charged or discharged upon touch or contact), or the like. Alternatively, one or more of the sensors 108-112 may be another type of sensor, such as a radar sensor, LIDAR sensor, etc.
The manipulator arm 114 is an elongated body of the robotic system 100 that can move in a variety of directions, grasp, and pull and/or push a brake rod. The controller 106 may be operably connected with the propulsion system 104 and the manipulator arm 114 to control movement of the robotic system 100 and/or the arm 114, such as by one or more wired and/or wireless connections. The controller 106 may be operably connected with the sensors 108-112 to receive data obtained, detected, or measured by the sensors 108-112.
The robotic system 100 can include a communication device 116 that communicates with an off-board control unit 118. The communication device 116 can represent one or more antennas and associated transceiving circuitry, such as one or more modems, transceivers, receivers, transmitters, etc. The control unit 118 can represent hardware circuitry that includes and/or is connected with one or more processors (e.g., microprocessors, field programmable gate arrays, or integrated circuits) that receives user input to remotely control movement and other operation of the robotic system 100. In one embodiment, the control unit 118 also represents one or more input devices. The input device may include at least one of a control column (e.g., joystick or steering wheel), a touch-sensitive screen, or an electronic switch (e.g., buttons or keys pressed by the operator). The input device may allow a user to remotely control movement and other operations of the robotic system 100. The control unit 118 also can include one or more antennas and associated transceiving circuitry to allow wireless communication with the communication device 116 of the robotic system 100. Alternatively or additionally, the communication device 116 of the robotic system 100 may be connected with the control unit 118 by one or more wired connections to allow for remote control of the robotic system 100 via the wired connection(s). For example, an operator may walk behind the robotic system holding a control unit that is wired to the robotic system and engage an input device to control operation of the robotic vehicle.
In one embodiment, the controller and/or the robotic control system may have a local data collection system deployed that may use machine learning to enable derivation-based learning outcomes. The controller or robotic control system may learn from and make decisions on a set of data (including data provided by the various sensors), by making data-driven predictions and adapting according to the set of data. In embodiments, machine learning may involve performing a plurality of machine learning tasks by machine learning systems, such as supervised learning, unsupervised learning, and reinforcement learning. Supervised learning may include presenting a set of example inputs and desired outputs to the machine learning systems. Unsupervised learning may include the learning algorithm structuring its input by methods such as pattern detection and/or feature learning. Reinforcement learning may include the machine learning systems performing in a dynamic environment and then providing feedback about correct and incorrect decisions. In examples, machine learning may include a plurality of other tasks based on an output of the machine learning system. In examples, the tasks may be machine learning problems such as classification, regression, clustering, density estimation, dimensionality reduction, anomaly detection, and the like. In examples, machine learning may include a plurality of mathematical and statistical techniques. In examples, the many types of machine learning algorithms may include decision tree based learning, association rule learning, deep learning, artificial neural networks, genetic learning algorithms, inductive logic programming, support vector machines (SVMs), Bayesian network, reinforcement learning, representation learning, rule- based machine learning, sparse dictionary learning, similarity and metric learning, learning classifier systems (LCS), logistic regression, random forest, K-Means, gradient boost, K-nearest neighbors (KNN), a priori algorithms, and the like. In embodiments, certain machine learning algorithms may be used (e.g., for solving both constrained and unconstrained optimization problems that may be based on natural selection). In an example, the algorithm may be used to address problems of mixed integer programming, where some components restricted to being integer-valued. Algorithms and machine learning techniques and systems may be used in computational intelligence systems, computer vision, Natural Language Processing (NLP), recommender systems, reinforcement learning, building graphical models, and the like. In an example, machine learning may be used for vehicle performance and behavior analytics, and the like.
In one embodiment, the controller and/or the robotic control system may include a policy engine that may apply one or more policies. These policies may be based at least in part on characteristics of a given item of equipment or environment. With respect to control policies, a neural network can receive input of a number of environmental and task-related parameters. These parameters may include an identification of a determined trip plan for a vehicle group, data from various sensors, and location and/or position data. The neural network can be trained to generate an output based on these inputs, with the output representing an action or sequence of actions that the vehicle group should take to accomplish the trip plan. During operation of one embodiment, a determination can occur by processing the inputs through the parameters of the neural network to generate a value at the output node designating that action as the desired action. This action may translate into a signal that causes the vehicle and/or robotic system to operate. This may be accomplished via back-propagation, feed forward processes, closed loop feedback, or open loop feedback. Alternatively, rather than using backpropagation, the machine learning system of the controller may use evolution strategies techniques to tune various parameters of the artificial neural network. The controller may use neural network architectures with functions that may not always be solvable using backpropagation, for example functions that are non-convex. In one embodiment, the neural network has a set of parameters representing weights of its node connections. A number of copies of this network are generated and then different adjustments to the parameters are made, and simulations are done. Once the output from the various models are obtained, they may be evaluated on their performance using a determined success metric. The best model is selected, and the vehicle controller executes that plan to achieve the desired input data to mirror the predicted best outcome scenario. Additionally, the success metric may be a combination of the optimized outcomes, which may be weighed relative to each other.
The controller and/or robotic control system can use this artificial intelligence or machine learning to receive input (e.g., a location or change in location), use a model that associates locations with different operating modes to select an operating mode of the one or more functional devices of the robotic system, the power system, or the like, and then provide an output (e.g., the operating mode selected using the model). The controller may receive additional input of the change in operating mode that was selected, such as analysis of noise or interference in communication signals (or a lack thereof), operator input, or the like, that indicates whether the machine-selected operating mode provided a desirable outcome or not. Based on this additional input, the controller can change the model, such as by changing which operating mode would be selected when a similar or identical location or change in location is received the next time or iteration. The controller can then use the changed or updated model again to select an operating mode, receive feedback on the selected operating mode, change or update the model again, etc., in additional iterations to repeatedly improve or change the model using artificial intelligence or machine learning.
The controller 106 may operate using the method represented by the state diagram 200 to move the robotic system 100 between or among different locations (e.g., in vehicle yards or other locations) to perform tasks, such as maintenance, inspection, repair, etc., of the vehicles. The controller 106 may operate in different operational modes. One mode can be referred to as an autonomous navigation mode and another mode can be referred to as tele-operation mode. Operations performed or controlled by the controller 106 can be referred to herein as modules. The modules can represent different sets of functions performed by the same or different processors of the controller 106, and/or can represent different hardware components (e.g., processors and associated circuitry) performing the functions associated with the respective modules.
At 202, the robotic system 100 is in a ready state. The ready state may involve the robotic system 100 being stationary and prepared to begin movement. The controller 106 may monitor the communication device 116 (or wait for a signal from the communication device 116) to indicate whether the robotic system 100 is to begin movement. The controller 106 may receive an input signal from the control unit 118 via the communication device 116 and/or from an input device of the robotic system 100 (e.g., one or more buttons, knobs, switches, touchscreens, keyboards, etc.). Responsive to receiving the input signal, the controller 106 may determine whether the input signal indicates that the robotic system 100 is to operate in the autonomous navigation mode (also referred to as “Autonomous NAV” in
If the controller 106 determines that the robotic system 100 is to operate in the tele-operational mode 203, then flow of the method or state diagram 200 may proceed toward 204. If the controller 106 determines that the robotic system 100 is to operate in the autonomous navigation mode 201, then flow of the method or state diagram 200 may proceed toward 208.
At 204, the robotic system 100 determines if a permissive signal to move has been generated or provided. The permissive signal may be generated or provided by a deliberation module of the controller 106. The deliberation module receives input from the control unit 118, such as movement of a joystick or other input that indicates a direction of movement, speed, and/or acceleration of the robotic system 100. The deliberation module of the controller 106 also examines data or other information provided by one or more of the sensors 108-112 to determine whether movement, as requested or dictated by the input received from the control unit 118, is feasible and/or safe. For example, the deliberation module of the controller 106 can obtain two dimensional (2D) image data (e.g., 2D images or video) from the sensors 109 and/or 111, three dimensional (3D) image data (e.g., 3D images or video, point clouds, etc.) from the sensors 108 and/or 110, and/or detection of engagement or touch of an object from the sensor 112. The deliberation module can examine this data to determine if the movement requested by the control unit 118 can be performed without the robotic system 100 colliding with another object or operating in another unsafe manner. For example, the deliberation module can examine the 2D and/or 3D image data to determine if one or more obstacles remain in the movement path requested by the input. As described in more detail below, the image data provided by one or more of the sensors 108-111 can be used to determine whether any objects are in the path of the robotic system 100.
Alternatively or in addition to the above, the deliberation module may use other data to navigate the robotic system within the external environment. For example, alternatively or in addition to an image sensor (e.g., camera), the robotic system may include other sensors configured to obtain data of the external environment. For example, the sensors may include one or more infrared sensors, one or more ultrasonic sensors, one or more lasers, one or more accelerometers, or one or more gyroscopes. Optionally, the deliberation module can communicate with an off-board controller that communicates data of the external environment. In some embodiments, the off-board controller may communicate movement authorities to the robotic system. Optionally, the off-board controller is part of a positive train control (PTC) system.
Data from one type of sensor may be used in conjunction with data from another type of sensor to determine location and heading of the robotic system. For example, the robotic system may use light detection and ranging (LIDAR) to determine at least one of a location of the robotic system within the external environment or a location of one or more objects within the external environment. The data obtained from LIDAR may be combined with other data for mapping the external environment.
Alternatively or in addition to LIDAR, the robotic system may utilize a laser guidance system. For example, the external environment in which the robotic system is configured to move may be mapped and stored onboard. The external environment may include designated reference points, such as reflective targets. Lasers mounted to the robotic system may emit optical signals that are reflected back toward the robotic system. One or more sensors mounted onto the robotic system may detect the reflected optical signals. Distances between the robotic system and the reference points may be determined and used to locate the robotic system within the environment.
In some embodiments, the sensors may include one or more accelerometers to detect linear acceleration along one or more axes and/or one or more gyroscopes to detect rotational rate. The accelerometer data and/or the gyroscopic data may be combined with other data for determining an orientation and location of the robotic system within the external environment.
For example, the robotic system may include an inertial measurement unit (IMU) or an attitude and heading reference system (AHRS). IMUs and AHRSs can include one or more accelerometers to detect linear acceleration and one or more gyroscopes to detect rotational rate. Optionally, the IMUs and AHRSs include a magnetometer that may be used to determine a heading reference. In particular embodiments, the IMUs and AHRSs include an accelerometer, a gyroscope, and a magnetometer for each of three axes that are perpendicular to one another. These axes may be referred to as pitch, roll, and yaw. Data from IMUs or AHRSs may be used in conjunction with other data, such as image data and/or laser or LIDAR data, to determine a location of the robotic system within the external environment and a direction of movement within the external environment. AHRSs may include an onboard processor for determining attitude and heading information.
Alternatively or in addition to the above, the robotic system may be guided by external elements within the environment that identify routes or paths. For example, magnetic tape may be positioned along the ground. One or more sensors mounted to the robotic system may be configured to detect the magnetic tape. The robotic system may follow the magnetic tape as the robotic system travels toward a destination. As another example, inductive wire may be embedded within the ground along one or more predetermined routes. The robotic system may include a sensor that detects a magnetic field generated by the inductive wire. The robotic system adjusts movement based on a strength of the magnetic field detected.
In some embodiments, the robotic system may include a locator device that determines a geographic locations of the robotic system and communicate with the deliberation module. The locator device may communicate with one or more location data sources that are off-board the robotic system to determine the locations of the robotic system. For example, the location data sources can represent GNSS satellites or beacons that broadcast signals that are received by the locator device (e.g., a GNSS or GPS receiver). The locator device can calculate the location, heading, speed, etc. of the vehicle based on these signals, as well as a confidence measurement of the location that is calculated from the received signals. Alternatively or in addition to location data, the locator device can include one or more sensors, such as those described above, that detect one or more characteristics to determine the locations of the robotic system. For example, the locator device can represent a reader that detects wireless signals or that optically scans machine-readable code. In particular, the locator device may include a reader that reads an RFID tag associated with a known location to determine the vehicle location. As another example, the locator device can represent an optical sensor (e.g., image sensor or laser detector) that optically reads where the vehicle is located (e.g., from one or more signs, such as waypoints, road signs, etc.) or that detects reflected signals from lasers. In one example, the locator device (and/or the vehicle controller) can apply one or more filters to the signals received from the location data source(s), such as a Kalman filter.
A robotic control system controller “RCS Controller” may be located onboard the robotic system and communicate with the deliberation module. The RCS controller can communicate with an off-board component, such as a control system back-office system. The RCS controller and the back-office system can communicate with each other via a communication device which can represent hardware transceiving circuitry, such as a transceiver, modem, antenna, and the like. The back-office system also can include a communication device to allow for communication with the robotic system. The RCS controller can represent hardware circuity that includes and/or is connected with one or more processors. The RCS controller can communicate with the back-office system to report locations of the robotic system, moving speeds of the robotic system, headings or directions of movement of the robotic system, etc. The RCS controller also can receive directive signals from the back-office system. These signals can include movement restrictions, such as movement authorities that dictate where, when, and/or how the robotic system can move. The back-office system may also communicate with other systems within the external environment, such as stationary power systems or non-stationary power systems (e.g., vehicles). The movement restrictions communicated to the robotic system may be based on locations of the other systems within the external environment. As one example, the movement restrictions communicated to the robotic system may be based on locations of rail cars located within a vehicle yard.
In some embodiments, the back-office system and the RCS controller can be components of a positive control system that sends movement authorities to the robotic system. The movement authorities may inform the robotic system whether the robotic system can travel into a designated region of the external environment, what speed to move in the designated region, etc. Optionally, the RCS controller can allow the robotic system to move into a designated region responsive to receiving the movement authority. In some instances, if the RCS controller does not receive the movement authority, then the RCS controller may generate signals to automatically control the propulsion system and/or braking system and/or steering system of the robotic system to prevent the robotic system from moving within the designated region in a manner that violates the movement authority (e.g., moving faster than the movement authority dictates).
As another example, the back-office system and the RCS controller can be components of a negative control system that sends movement authorities to robotic systems to inform the robotic system where the robotic system cannot move. If the RCS controller receives a movement authority from the back-office system indicating that the robotic system cannot enter into the designated region, then the RCS controller can automatically control the propulsion system and/or braking system and/or steering system to prevent disallowed movement of the robotic system. If the RCS controller does not receive the movement authority from the back-office system indicating that the robotic system cannot enter into the designated region, then the RCS controller may not allow movement of the vehicle in the designated region.
If the controller 106 (e.g., the deliberation module) determines that the robotic system 100 can move according to the input provided by the control unit 118 at 202, then flow of the method or state diagram 200 continues toward 206. Otherwise, the method or state diagram 200 may remain at 204 until permission to move is received from or otherwise provided by the deliberation module of the controller 106.
At 206, the robotic system 100 moves according to the input provided by or otherwise received from the control unit 118. For example, responsive to receiving permission to move the robotic system 100 according to the input provided by the control unit 118, the controller 106 may generate control signals that are communicated to the propulsion system 104 of the robotic system 100 to move the robotic system 100 according to the input. Upon completion of the movement, the propulsion system 104 may stop moving the robotic system 100 and flow of the method or state diagram 200 may return toward 204.
If, at 204, it is determined that the robotic system 100 is to operate in the autonomous navigation mode 201, then flow of the method or state diagram 200 may proceed toward 208. For example, if the input received by the controller 106 from the control unit 118 indicates that the robotic system 100 is to autonomously move, then flow may proceed toward 208.
At 208, a navigation module of the controller 106 informs the deliberation module that autonomous movement of the robotic system 100 has been initiated. This can involve the controller 106 from the manual navigation mode to the autonomous navigation mode. The robotic system 100 may remain stationary and optionally prohibit movement of the robotic system 100 until confirmation of the change from the manual to autonomous navigation mode has been received. This confirmation may be provided from the deliberation module of the controller 106.
Responsive to receiving confirmation that the autonomous movement of the robotic system 100 has been initiated, at 210, a determination is made as to whether the robotic system 100 can move. This determination may involve examining data provided by one or more of the sensors 108-112, in addition to or exclusive of other data provided to or accessible by the controller 106. For example, in addition to the image data provided by one or more of the sensors 108-111, the controller 106 may access a memory or database (not shown) onboard or off-board the robotic system 100 (e.g., via the communication device 116). The controller 106 can obtain information such as a current location of the robotic system 100 (e.g., via a global positioning system receiver or data), locations of vehicles in the vehicle yard, numbers of vehicles in a vehicle consist that the robotic system 100 is to move alongside, known or designated locations of objects in or around the robotic system 100, etc.
A perception module of the controller 106 can examine the sensor data and/or other data to determine how to autonomously move the robotic system 100. The perception module can examine this data to determine how to safely and efficiently move the robotic system 100 without intervention (or at least additional intervention) from a human operator.
Returning to the description of the method or state diagram 200 shown in
At 212, the robotic system 100 autonomously moves. In one embodiment, the navigation module of the controller 106 determines a waypoint location for the robotic system 100 to move toward. The waypoint location may be a geographic location that is between a current location of the robotic system 100 and a final, destination, or goal location that the robotic system 100 is moving toward. For example, if the robotic system 100 is to move five meters to a brake lever of a vehicle in order to grasp and pull the brake lever (e.g., to bleed an air brake of the vehicle), the navigation module may generate control signals to cause the robotic system 100 to move to a waypoint that is fifty centimeters (or another distance) toward the brake lever from the current location of the robotic system 100, but that is not at the location of the brake lever.
In one embodiment, the controller 106 can determine the locations of objects using the data with simultaneous localization and mapping (SLAM). For example, the controller 106 can use real-time appearance-based mapping (RTAB-Map) to identify the locations of objects.
The navigation module of the controller 106 can generate control signals to dictate how the robotic system 100 moves toward the waypoint location. These control signals may designate the direction of movement, the distance that the robotic system 100 is to move, the moving speed, and/or acceleration based on the current location of the robotic system 100, the waypoint location, and/or limitations determined by the perception module of the controller 106 (described below). The navigation module generates control signals that are communicated to the propulsion system 104 of the robotic system 100. These control signals direct the motors and other components of the propulsion system 104 how to operate to move the robotic system 100.
Movement of the robotic system 100 can be monitored to determine whether the movement of the robotic system 100 has or will violate one or more predefined or previously designated limits. In the illustrated example, the robotic system 100 is not allowed to move more than forty inches (e.g., 102 centimeters). Optionally, another distance limitation or other limitation (e.g., a limitation on an upper or lower speed, a limitation on an upper or lower acceleration, a limitation on a direction of movement, etc.) may be used. If the movement of the robotic system 100 reaches or violates one or more of these limitations, flow of the method or state diagram 200 can proceed toward 214.
In one embodiment, the controller 106 determines the movements of the robotic system 100 to try and achieve different goals. One goal is to move the robotic system 100 so as to minimize or reduce the distance between the robotic system 100 and the desired location, such as the next waypoint (relative to moving the robotic system 100 along one or more, or all, other feasible paths to the next waypoint). Another goal is to keep at least a designated safe distance between the robotic system 100 and one or more other objects, such as rail tracks on which the vehicles are disposed. The controller 106 can determine commands for the propulsion system 104 that drive the robotic system 100 toward the next waypoint and fuse these commands with commands that keep the robotic system 100 away from the vehicles (or other objects), by at least a designated, non-zero distance (e.g., four inches or ten centimeters). These commands are combined by the controller 106 to determine a velocity command that will control the propulsion system 104 of the robotic system 100 to move. The fusion can be a weighted sum of the commands:
cmdvel=α*cmdgoal+β*cmdsafety (1)
α+β=1 (2)
where cmdvel represents the velocity command, cmdgoal and cmdsafety are generated using the artificial potential field algorithm, and α and β are parameters that are tuned or set based on the task-relevant situations.
At 214, movement of the robotic system 100 is stopped. The navigation module of the controller 106 can generate and communicate an alarm signal to the propulsion system 104 that stops movement of the robotic system 100. This signal can direct motors to stop rotating wheels of the vehicle 102 of the robotic system 100 and/or direct a brake of the vehicle 102 to stop movement of the robotic system 100. Flow of the method or state diagram 200 may then return toward 202.
But, if movement of the robotic system 100 does not reach or violate the limitation(s), then the robotic system 100 may continue autonomously moving toward the waypoint location. The following motion will be determined through a decision making process (e.g., motion and energy optimization). As described above, the navigation module may direct the propulsion system 104 to move the robotic system 100 to move toward, but not all the way to, a destination or goal location. Instead, the navigation module can direct the propulsion system 104 to move the robotic system 100 part of the way to the destination or goal location.
The robotic system 100 moves toward the waypoint location subject to the limitations described above. Upon reaching the waypoint location, flow of the method or state diagram 200 can return toward 210 from 212. For example, the perception module can again determine whether the robotic system 100 can move based on the sensor data and/or other data, as described above. At least a portion of the method or state diagram 200 may repeat one or more times or iterations between perceiving the surroundings, determining a subsequent waypoint, determining limitations on movement toward the waypoint, and moving to the waypoint. Eventually, the final waypoint may be at or near the final destination of the robotic system 100.
In one or more embodiments, a system is provided that includes a robotic vehicle having a propulsion system configured to propel the robotic vehicle between different final destinations and a manipulator configured to perform designated tasks. The system also includes one or more sensors disposed onboard the robotic vehicle configured to obtain image data representative of an external environment and to sense operation of the manipulator. The system also includes a local controller disposed onboard the robotic vehicle and configured to receive input signals from an off-board controller. Responsive to receiving an input signal for moving in an autonomous mode, the local controller is configured to move the robotic vehicle toward one of the different final destinations by autonomously and iteratively determining a series of waypoints until the robotic vehicle has reached the one final destination. For each iteration, the local controller is configured to determine a next waypoint between a current location of the robotic vehicle and the final destination. The local controller is also configured to determine movement limitations of the robotic vehicle for moving the robotic vehicle between the current location and the next waypoint. The movement limitations are based on the image data of the external environment including the current location and the next waypoint. The movement limitations are configured to avoid collisions between the robotic vehicle and one or more objects in the external environment. The local controller is also configured to generate control signals in accordance with the movement limitations that are configured to move the robotic vehicle toward the next waypoint. Responsive to receiving the input signal for operating in a tele-operation mode, the local controller is configured to exit the autonomous mode.
Optionally, the input signal for operating in the tele-operation mode may include a remote command that dictates at least one of a movement of the robotic vehicle or a movement of the manipulator. The local controller is configured to determine whether the remote command is permissible, wherein, responsive to determining that the remote command is permissible, the local controller is configured to generate control signals for performing the remote command.
Optionally, the local controller is configured to withhold issuing the control signals for performing the remote command until the local controller determines that the remote command is permissible.
Optionally, the remote command may include at least one of a direction of movement, a speed, or an acceleration.
Optionally, the one or more objects can include movable objects that travel along a pathway. The movement limitations restrict movement of the robotic vehicle to at least a designated distance away from the pathway. For example, the movable objects may include vehicles (e.g., rail cars).
Optionally, the pathway is a fixed pathway, such as rail tracks on which the vehicles are disposed, and the final destinations are adjacent to the movable objects along the fixed pathway. The local controller is also configured to generate control signals for controlling the manipulator to perform a task related to the movable objects.
Optionally, the local controller is also configured to identify the final destinations based on the image data. The final destination can include a task object for the manipulator to engage.
Optionally, the local controller is configured to determine the movement limitations using simultaneous localization and mapping (SLAM) of the image data.
Optionally, the system also may include the off-board controller. The off-board controller may include an input device configured to be manipulated by a user of the off-board controller.
Optionally, the off-board controller is connected to the local controller via a wired connection and communicates to the local controller through the wired connection.
In one or more embodiments, a method is provided that may include obtaining, at a local controller disposed onboard a robotic vehicle, image data representative of an external environment of the robotic vehicle. The external environment may include a designated region where the robotic vehicle performs tasks. For example, the tasks may be performed at different final destinations within the designated region. Responsive to receiving an input signal for moving the robotic vehicle in an autonomous mode, the method further may include iteratively generating, by the local controller, control signals for moving the robotic vehicle through a series of waypoints toward one of the different final destinations, wherein cach of the iterations may include determining a next waypoint between a current location of the robotic vehicle and the final destination. Each iteration also may include determining movement limitations of the robotic vehicle for moving the robotic vehicle between the current location and the next waypoint. The movement limitations are based on the image data of the external environment including the current location and the next waypoint. The movement limitations are configured to avoid collisions between the robotic vehicle and one or more objects in the external environment. Each iteration also may include generating control signals in accordance with the movement limitations that are configured to move the robotic vehicle toward the next waypoint. Responsive to receiving an input signal for operating in a tele-operation mode, the method further comprises exiting the autonomous mode.
Optionally, the input signal for operating in the tele-operation mode may include a remote command that dictates at least one of a movement of the robotic vehicle or a movement of a manipulator of the robotic vehicle. The method also may include determining whether the remote command is permissible, wherein, responsive to determining that the remote command is permissible, generating control signals for performing the remote command.
Optionally, the control signals for performing the remote command are withheld until the remote command is determined to be permissible.
Optionally, the remote command may include at least one of a direction of movement, a speed, or an acceleration.
Optionally, the one or more objects can include movable objects that travel along a pathway. The movement limitations restrict movement of the robotic vehicle to at least a designated distance away from the pathway.
Optionally, the waypoints are at most fifty centimeters apart.
Optionally, the method also may include identifying the final destinations based on the image data. The final destinations can include a task object for a manipulator of the robotic vehicle to engage.
Optionally, determining the movement limitations may include using simultaneous localization and mapping (SLAM) of the image data.
Optionally, the method also may include performing a task at the one final destination using a manipulator of the robotic vehicle.
Optionally, the movement limitations for moving toward different waypoints can include at least one of a different direction of movement, a different speed, or a different acceleration of the robotic vehicle.
In one embodiment, a robotic system includes a robotic vehicle having a propulsion system configured to propel the robotic vehicle, one or more sensors configured to be disposed onboard the robotic vehicle and to obtain image data representative of an external environment, and a controller configured to be disposed onboard the robotic vehicle and to determine a waypoint for the robotic vehicle to move toward. The waypoint is located between a current location of the robotic vehicle and a final destination of the robotic vehicle. The controller also is configured to determine limitations on movement of the robotic vehicle toward the waypoint. The limitations are based on the image data. The controller is configured to control the propulsion system to move the robotic vehicle to the waypoint subject to the limitations on the movement to avoid colliding with one or more objects. The controller also is configured to determine one or more additional waypoints subsequent to the robotic vehicle reaching the waypoint, determine one or more additional limitations on the movement of the robotic vehicle toward each of the respective additional waypoints, and control the propulsion system of the robotic vehicle to sequentially move the robotic vehicle to the one or more additional waypoints.
In one example, the one or more sensors are configured to obtain a point cloud of the external environment using one or more structured light sensors.
In one example, controller is configured to determine the limitations on the movement of the robotic vehicle by determining relative locations of one or more objects in the external environment based on the image data and restricting movement of the robotic vehicle to avoid colliding with the one or more objects.
In one example, the controller is configured to determine the limitations using simultaneous localization and mapping to restrict the movement of the robotic vehicle.
In one example, the controller also is configured to stop movement of the robotic vehicle responsive to the robotic vehicle moving farther than a designated, non-zero distance toward the waypoint.
In one example, the final destination of the robotic vehicle is a brake lever of a vehicle.
In one example, the controller also is configured to switch between manual control of the movement of the robotic vehicle and autonomous movement of the robotic vehicle based on input received from a control unit disposed off-board the robotic vehicle.
In one embodiment, a method includes obtaining image data representative of an environment external to a robotic system, determining a waypoint for the robotic system to move toward, the waypoint located between a current location of the robotic system and a final destination of the robotic system, determining limitations on movement of the robotic system toward the waypoint. The limitations are based on the image data, controlling a propulsion system of the robotic system to move the robotic system to the waypoint subject to the limitations on the movement to avoid colliding with one or more objects, determining one or more additional waypoints subsequent to the robotic system reaching the waypoint, determining one or more additional limitations on the movement of the robotic system toward each of the respective additional waypoints, and controlling the propulsion system of the robotic system to sequentially move the robotic system to the one or more additional waypoints.
In one example, obtaining the image data includes obtaining a point cloud using one or more structured light sensors.
In one example, determining the limitations on the movement of the robotic system include determining relative locations of one or more objects in the environment based on the image data and restricting movement of the robotic system to avoid colliding with the one or more objects.
In one example, determining the limitations includes using simultaneous localization and mapping to restrict the movement of the robotic system.
In one example, the method also includes stopping movement of the robotic system responsive to the robotic system moving farther than a designated, non-zero distance toward the waypoint.
In one example, the final destination of the robotic system is a brake lever of a vehicle.
In one example, the method also includes switching between manual control of the movement of the robotic system and autonomous movement of the robotic system based on input received from a control unit disposed off-board the robotic system.
In one embodiment, a robotic system includes a robotic vehicle having a propulsion system configured to propel the robotic vehicle, one or more structured light sensors configured to be disposed onboard the robotic vehicle and to obtain point cloud data representative of an external environment, and a controller configured to be disposed onboard the robotic vehicle and to determine a waypoint for the robotic vehicle to move toward. The waypoint is located between a current location of the robotic vehicle and a brake lever of a rail vehicle. The controller also is configured to determine limitations on movement of the robotic vehicle toward the waypoint. The limitations are based on the point cloud data. The controller is configured to control the propulsion system to move the robotic vehicle to the waypoint subject to the limitations on the movement to avoid colliding with one or more objects.
In one example, the controller also is configured to determine one or more additional waypoints subsequent to the robotic vehicle reaching the waypoint, determine one or more additional limitations on the movement of the robotic vehicle toward each of the respective additional waypoints, and control the propulsion system of the robotic vehicle to sequentially move the robotic vehicle to the one or more additional waypoints.
In one example, the controller is configured to determine the limitations on the movement of the robotic vehicle by determining relative locations of one or more objects in the external environment based on the point cloud data and restricting movement of the robotic vehicle to avoid colliding with the one or more objects.
In one example, the controller is configured to determine the limitations using simultaneous localization and mapping to restrict the movement of the robotic vehicle.
In one example, the controller also is configured to stop movement of the robotic vehicle responsive to the robotic vehicle moving farther than a designated, non-zero distance toward the waypoint.
In one example, the controller also is configured to switch between manual control of the movement of the robotic vehicle and autonomous movement of the robotic vehicle based on input received from a control unit disposed off-board the robotic vehicle.
In one embodiment, a system is provided that includes a robotic vehicle having a propulsion system configured to propel the robotic vehicle through an external environment and an actuator configured to perform designated operations while in the external environment. The system also includes one or more sensors disposed onboard the robotic vehicle configured to obtain environmental data representative of the external environment. The system also includes a local controller disposed onboard the robotic vehicle and configured to receive input signals from an off-board controller. Responsive to receiving an input signal from the off-board controller for moving in an autonomous mode, the local controller is configured to autonomously move the robotic vehicle within the external environment. Responsive to receiving an input signal for operating in a tele-operation mode, the local controller is configured to exit the autonomous mode, wherein the input signal for operating in the tele-operation mode includes a remote command that dictates at least one of a movement of the robotic vehicle or a movement of the actuator.
In one example, the local controller is configured to determine whether the remote command is permissible, wherein, responsive to determining that the remote command is permissible, the local controller is configured to generate control signals for performing the remote command. Optionally, the local controller is configured to withhold issuing the control signals for performing the remote command until the local controller determines that the remote command is permissible.
In one example, the remote command includes at least one of a direction of movement that differs from the current direction of movement, a speed that differs from the current speed, or an acceleration that differs from the current acceleration.
In one example, the local controller is configured to autonomously and repeatedly determine a current location of the robotic vehicle within the external environment based on the environmental data. The local controller is configured to move the robotic vehicle within the external environment based on the current location of the robotic vehicle and a designated destination.
In one example, the system also includes the off-board controller. The off-board controller includes an input device configured to be controlled by a user of the off-board controller.
In one example, the input device includes at least one of a control column, a touch-sensitive screen, or an electronic switch.
In one example, the off-board controller is connected to the local controller via a wired connection and communicates to the local controller through the wired connection.
In one example, the system also includes a control device that includes the off-board controller. The control device is a handheld device carried by a user.
In one example, the environmental data includes at least one of image data, light detection and ranging (LIDAR) data, laser guidance data, accelerometer data, gyroscopic data, data from an inertial measurement unit (IMU), data from an attitude and heading reference system (AHRS).
In one embodiment, a method is provided that includes obtaining, at a local controller disposed onboard a robotic vehicle, environmental data representative of an external environment of the robotic vehicle. The external environment includes a designated region where the robotic vehicle moves and performs operations. Responsive to receiving an input signal for moving the robotic vehicle in an autonomous mode, the method also includes autonomously generating control signals, by the local controller, for moving the robotic vehicle through the external environment. Responsive to receiving an input signal for operating in a tele-operation mode, the method further comprises exiting the autonomous mode, wherein the input signal for operating in the tele-operation mode includes a remote command that dictates at least one of a movement of the robotic vehicle or a movement of the actuator.
In one example, the method also includes determining whether the remote command is permissible, wherein, responsive to determining that the remote command is permissible, generating control signals for performing the remote command.
In one example, the remote command includes at least one of a direction of movement, a speed, or an acceleration that differs from a direction of movement, a speed, or an acceleration of the robotic vehicle when exiting the autonomous mode.
In one example, the current location is based on at least one of location data from a satellite or beacon or a predetermined map representative of at least a portion of the external environment.
In one example, the method also includes receiving a user command at the off-board controller. The user command is received from an input device controlled by a user of the off-board controller.
As used herein, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural of said elements or steps, unless such exclusion is explicitly stated. Furthermore, references to “one embodiment” of the presently described subject matter are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. Moreover, unless explicitly stated to the contrary, embodiments “comprising” or “having” an element or a plurality of elements having a particular property may include additional such elements not having that property.
It is to be understood that the above description is intended to be illustrative, and not restrictive. For example, the above-described embodiments (and/or aspects thereof) may be used in combination with each other. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the subject matter set forth herein without departing from its scope. While the dimensions and types of materials described herein are intended to define the parameters of the disclosed subject matter, they are by no means limiting and are exemplary embodiments. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the subject matter described herein should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects. Further, the limitations of the following claims are not written in means-plus-function format and are not intended to be interpreted based on 35 U.S.C. § 112(f), unless and until such claim limitations expressly use the phrase “means for” followed by a statement of function void of further structure.
This written description uses examples to disclose several embodiments of the subject matter set forth herein, including the best mode, and also to enable a person of ordinary skill in the art to practice the embodiments of disclosed subject matter, including making and using the devices or systems and performing the methods. The patentable scope of the subject matter described herein is defined by the claims, and may include other examples that occur to those of ordinary skill in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.
This application is a continuation-in-part of U.S. patent application Ser. No. 16/934,046 (U.S. Pat. Pub. No. 2020-0348686) (hereinafter “the '046 Application”), filed on 21 Jul. 2020, which is a continuation of 15/282, 102 (U.S. Pat. Pub. No. 2017-0341235), which was filed on 30 Sep. 2016 and claims priority to U.S. Provisional Application No. 62/342,448, filed on 27 May 2016, the entire disclosure of each is incorporated herein by reference. The '046 Application is also a continuation-in-part of U.S. patent application Ser. No. 16/240,237 (U.S. Pat. Pub. No. 2019-0134821) (hereinafter “the '237 Application”), which was filed on Jan. 4, 2019 and which is a continuation-in-part of U.S. patent application Ser. No. 15/292,605 (U.S. Pat. Pub. No. 2017-0341236), which was filed on Oct. 13, 2016 and which claims priority to U.S. Provisional Application No. 62/342,510, filed May 27, 2016, the entire disclosure of each is incorporated herein by reference. The '237 Application is also a continuation-in-part of and claims priority to U.S. patent application Ser. No. 15/885,289, filed Jan. 31, 2018, and is titled “Systems and Methods for Control of Robotic Manipulation” that claims priority to, and is a continuation of, U.S. patent application Ser. No. 14/702,014 (now U.S. Pat. No. 9,889,566), filed 1 May 2015, and entitled “Systems and Methods for Control of Robotic Manipulation,” the entire subject matter of each is incorporated herein by reference. The '237 Application is also a continuation-in-part of and claims priority to U.S. patent application Ser. No. 15/058,560 (U.S. Pat. Pub. No. 2017-0173790, filed Mar. 2, 2016, and is titled “Control System and Method for Applying Force to Grasp a Brake Lever”, that claims priority to U.S. Provisional Application Nos. 62/269,523; 62/269,425; 62/269,377; and 62/269,481, all of which were filed on 18 Dec. 2015, and the entire disclosure of each is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62342448 | May 2016 | US | |
| 62342510 | May 2016 | US | |
| 62269523 | Dec 2015 | US | |
| 62269425 | Dec 2015 | US | |
| 62269377 | Dec 2015 | US | |
| 62269481 | Dec 2015 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15282102 | Sep 2016 | US |
| Child | 16934046 | US | |
| Parent | 14702014 | May 2015 | US |
| Child | 15885289 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16934046 | Jul 2020 | US |
| Child | 18582804 | US | |
| Parent | 16240237 | Jan 2019 | US |
| Child | 16934046 | US | |
| Parent | 15292605 | Oct 2016 | US |
| Child | 16240237 | US | |
| Parent | 15885289 | Jan 2018 | US |
| Child | 16240237 | US | |
| Parent | 15058560 | Mar 2016 | US |
| Child | 16240237 | US |
