Patent Application
20230123504
The present disclosure relates to systems and methods for controlled cleaning of vehicles, and in particular, for controlled cleaning of the interior of a vehicle.
Internal cleaning of passenger vehicle operations at car wash locations have typically been the domain of manual laborers. This is due to the inherent high mix, dynamic environment for which these operations occur. Though external passenger vehicle washing has seen pretty standard implementations of automated solutions, the interior provides additional challenges. The interiors vary greatly across vehicles, they contain many more confined/tight spaces, and involve numerous different processes that often require wiping with a cloth, which involves a lot of force feedback and dexterity. Further complicating matters is that the presentation of the passenger vehicle is not often repeatable, so it may be difficult for legacy automation approaches to know with confidence where the features/surfaces to be processed are located in space, not to mention managing the different items that may be left in a vehicle that need to be managed or planned around or avoided. These and other deficiencies exist.
Embodiments of the present disclosure provide a system for cleaning an interior of a vehicle. The system may include a robotic arm positioned outside of the vehicle. The robotic arm may include an end effector configured as a cleaning implement for cleaning a surface in the interior of the vehicle. The system may include a first camera configured to determine a position of the vehicle with respect to a reference point. The system may include a second camera configured to scan the interior of the vehicle. The system may include a first controller configured to create and/or modify a tool path to execute a cleaning operation, based on the scan, and to send instructions to the robotic arm to execute the cleaning operation in accordance with the created or modified tool path.
Embodiments of the present disclosure provide an automated method for cleaning an interior of a vehicle. The method may include determining a position of the vehicle with respect to one or more robotic arms positioned exterior to the vehicle. The method may include scanning a configuration of the vehicle to yield a configuration scan. The method may include identifying surfaces to be cleaned. The method may include creating and/or modifying a plurality of tool paths for the one or more robotic arms to clean the identified surfaces. The method may include controlling the one or more robotic arms to move along the created and/or modified plurality of tool paths and execute a cleaning operation in the interior of the vehicle
Embodiments of the present disclosure provide automated method for cleaning an interior of a vehicle. The method may include storing a plurality of images for a plurality of vehicles in a database to create a master data package. The method may include acquiring vehicle specific data from the master data package. The method may include scanning a configuration of the vehicle. The method may include aligning the acquired vehicle specific data with the scanned configuration. The method may include creating a process plan for execution of a plurality of cleaning operations. The method may include sending instructions to a robot to execute the plurality of cleaning operations based on the process plan. The method may include executing the plurality of cleaning operations in accordance with the instructions.
The following description of embodiments provides non-limiting representative examples referencing numerals to particularly describe features and teachings of different aspects of the invention. The embodiments described should be recognized as capable of implementation separately, or in combination, with other embodiments from the description of the embodiments. A person of ordinary skill in the art reviewing the description of embodiments should be able to learn and understand the different described aspects of the systems and methods disclosed herein. The description of embodiments should facilitate understanding of the systems and methods to such an extent that other implementations, not specifically covered but within the knowledge of a person of skill in the art having read the description of embodiments, would be understood to be consistent with an application thereof.
The systems and methods disclosed herein are configured to provide controlled cleaning of vehicles, and in particular, the interior of a vehicle. The implementation of the systems and methods disclosed herein provides advances in robotics technology, including the ability to reconstruct surfaces, identify features/surfaces, localize and register items to be processed, autonomous tool path planning and process planning based on presented information, and motion planning that includes coordination of external axes, with high fidelity collision avoidance, and represents improvements over implementations that, in addition to the above-identified deficiencies, are typically low margin and that have historically relied on low cost labor to realize the requirements of the internal cleaning operations. The solution described herein provides a guided autonomous solution, where a human may or may not be guiding the process, either quite minimally, or more so, depending on complexity and other considerations, to realize the idea of a reduction of manual touch labor that currently occurs on vehicles that enter a car wash bay, thereby reducing operational costs, and improving automobile throughput at high levels of quality. The systems and methods described herein are configured to meet rigorous throughput and cost requirements of the vehicle cleaning industry by employing a safe, effective, and efficient cleaning process within the footprint of standard vehicle wash facilities.
Also included in the solution, and in accordance with the systems and methods described herein, is the mounting, tooling, and/or fitting of the robots. In particular, one or more robots may be used to perform one or more commands, such as cleaning operations of the interior of a vehicle. The one or more robots may be positioned overhead and/or can be floor mounted. According to one embodiment, a robot can be mounted overhead and positioned to clear the open doors of a vehicle, and still be able to reach a floor of the vehicle. Without limitation and by way of example, a vehicle may refer to a car or a truck. Conventional robots, such as a Yaskawa HC20 robot with a 1700 mm reach may not be suitable. For instance, if this robot is mounted overhead at a height that would allow vehicles to pass safely below it, it would not be possible for the robot to reach the floor. Accordingly, a 2-axis gantry with a collaborative robot can be utilized. In particular, a 2-axis gantry system that is configured to both track and extend, vertically for a ceiling mounted robot, and horizontally for a floor mounted robot can be utilized. In some examples, this would extend far enough into the ceiling or wall so as to be out of the way of the vehicle, and out far enough to position the robot closer to the vehicle. In some examples, a larger non-collaborative robot, such as Yaskawa MH50 II-20, would allow the robot to reach into the vehicle, and for the vehicle to safely pass under it. In addition, the robot may be configured to reach into the footwell at a sufficient angle to avoid collision with the door of the vehicle. It is to be appreciated that other suitable configurations, such as a scara-type rotating linkage may also be employed to extend reach into a vehicle while keeping the robot out of the way. In addition, it is to be further appreciated that autonomous approaches (without limitation, through implementation of a simultaneous localization and mapping (SLAM) algorithm) may be configured to scan the interior of the vehicle and plan the trajectory of the tool path(s) in real-time, or in which autonomous robot manipulators may be configured to map out the interior of the vehicle to perform the cleaning of the interior of the vehicle.
A “Master Scan” process can be conducted prior to the vehicle arriving at a facility, where a technician captures a high-resolution 3D scan of the vehicle's interior, in which the scan may be processed to produce a model-specific Master Scan Data Package (“MSDP”). Without limitation, this data package may include a file or plurality of files including 3D mesh models, process tool paths, and various other metadata. The MSDP is transmitted to a database 102, such as a local database, at the facility. This information can be transmitted to local databases at a plurality of facilities within a network, or can be uploaded to a cloud or cloud-based server in a computing environment and deployed via one or more containers so that each facility can pull information when needed. Thus, when the vehicle arrives at the facility, the vehicle is identified, such as through a license plate or VIN scanner, to determine a specific model of the vehicle as stored in the database 102. Alternatively, the MSDP creation process can occur at the facility just ahead of the cleaning operation. This process can be used for a vehicle that has not been seen before based on a set of rules relative to the generalizability of the data in the one or more databases relative to available master scan data packages. Additionally, the MSDP may include segmented features, such as consoles, door scans, and seats. The MSDP creation process may leverage a 3D scanner. The scanner collects data about various surfaces within the vehicle in order to recreate such surfaces of the vehicle to be processed during cleaning. In some examples, regarding car windows, the 3D scanner may be tilted to get into edges where the glass touches the door. At a minimum, the scanner must be configured to obtain the edges or corners of the windows of the vehicle.
The master scan may be configured to generate new or augment existing measurements for use in the creation of 3D maps of vehicle interiors, including obstructions which are not part of the originally produced vehicle. Preferable scanning method uses 3D time of flight cameras for high resolution and speed, but other scanning methods such as LIDAR or other available techniques can be used. Once captured, the scanned data may be transferred to and stored in a database 102 for retrieval by a robotic arm 110.
A VIN reader or scanning system may be configured to, as part of the vehicle identification, read VIN, or measure length, width, or wheelbase, or identify other distinguishing features of a vehicle for purpose of attributing the identifying characteristics to specific vehicle makes, years, and models. This data may be transmitted and stored in a database 102, which may contain interior and exterior information attributed to specific vehicle makes, years, and models and the database 102 may be configured to utilize this information to alert the robotic arms 110 to locate an interior 3D map for the relevant vehicle. The scanning system can be a handheld VIN reading system or linked to other camera-based systems, including but not limited to first camera 111 and/or second camera 112.
As depicted in
At Stage U, any number of passengers may be unloaded for vehicle preparation. At Stage A, a camera 111 may be configured to inspect the vehicle to determine its approximate position on a conveyor belt 117 (Coarse Localization 104). This camera 111 can be a 2D camera or 3D camera, positioned overhead relative to the vehicle, and can capture one or more images. This measurement allows the system 100 to adapt a “Configuration Scan” robot path from the database 102 to match the vehicle's position. Stage B can include two robot-mounted cameras 112, such as 3D or depth cameras, to acquire one or more additional images. These cameras 112 may be disposed on opposite sides of the vehicle, such as left and right sides of the vehicle. In some examples, the arrangement of robotic arms 110 or manipulators may be symmetric, whereas in other examples, the arrangement of the robotic arms 110 or manipulators may be asymmetric. Without limitation, the robotic arm 110 may be a part of a robot. The robotic arms 110 at Stage B execute the Configuration Scan robot paths to position the cameras, which capture Configuration Scan data of the vehicle interior from several viewpoints in both the front and rear compartments (Scan, Locate, Query 109). This vehicle-specific scan is processed and compared against the MSDP. Then, the coarse vehicle alignment from Stage A is fine-tuned and tool paths are shifted, such as to account for front seat adjustments (Configuration Alignment 105), as well as any obstacles that may be observed in the Configuration Scan data. These obstacles are accounted for by treating them as collision objects, and motion planning is updated to account for these obstacles. In some examples, the tool paths may simply be truncated in certain instances, or the free space available for robot motion simply updated by the presence of the observed obstacle. Tool paths from the MSDP are aligned to the current vehicle position, adjusted to avoid detected obstacles, and converted to Robot Motion Paths (Process Planning 106). These computed paths are transferred to robot controllers to execute robot motion and to control process tooling (Motion Execution 107) at Stage C. In the provided example, four robotic arms 110 are utilized for Motion Execution; however, any suitable number of robotic arms 110 can be employed. At Stage D, one or more operators can enter the process to complete a final manual touch-up and inspection. At this stage, the operator(s) can enter feedback into the system 100 to identify any areas for improvement by later automated and/or manual processes (QA Feedback”).
As shown in
In some examples, a robotic arm 110 may be part of an interconnected robot system. The robotic arm 110 may be positioned outside of a vehicle. The robotic arm 110 may include an end effector. The end effector may be configured as a cleaning implement for cleaning a surface within the vehicle. Without limitation, the surface within the vehicle may refer to an interior surface of the vehicle. In some examples, at least one of the robotic arm 110 and the end effector may include a sensor that is configured to detect objects that are present inside the vehicle. In some examples, a plurality of end effectors of the robotic arm 110 may be configured specifically for vehicle interior cleaning operations. Moreover, a plurality of sensors incorporated into the end effectors and/or robotic arms 110 may be configured to detect the position of objects and obstacles in the vehicle interior in a manner that is robust and timely enough to allow for in-situ collision avoidance via robot control software algorithms.
Regarding the interconnected robot system, this may be configured such that a plurality, including but not limited to up to 4-sets of two robotic arms 100 may be positioned outside the openings of various vehicles, and the vehicles may be conveyed to a point where the interconnected robot systems are stationed. Two robotic arms 110 may be joined and mounted on a custom-designed cantilevered, pivotable track system allowing for each robotic arm 110 pair to move up and down and reach horizontally to enter the vehicle and address roughly one quarter of the interior vehicle space to be cleaned. Multi-axis robotic arms 110 may be selected for their compactness and ability to easily enter and exit the vehicle through the door opening, as well as for their payload, reach, and dexterity once inside the vehicle. The interconnected robot systems can include any number of robotic arms 110, which may be configured to move at higher speeds, or comparably sized collaborative robots, which may be selected for their built-in safety features. The interconnected robot systems may be positionally fixed at a location such that vehicles move to the location of the interconnected robot systems in order to be cleaned and thereafter move away from the system 100. Alternatively, the interconnected robot system may be configured to move along with a vehicle as the vehicle moves along a predetermined travel pathway. The robotic arms 110 of the interconnected robot system can be positioned such that they are affixed on either side of a vehicle to be cleaned, or the robotic arms 110 can be positioned overhead of the vehicle. Each of the interconnected robot system may include a robot programmable logic controller and graphical interface which links robot control software of the robotic arm 110 to the vehicle wash facility's conveyor control system, allowing for sensor driven or robot control software-driven, or control of interior cleaning line conveyor speed, acceleration, and vehicle placement. In some examples, the graphic interface may be configured to monitor the performance by the system 100. For example, the graphic interface may indicate a score for assessing how the cleaning of the interior of the vehicle, performed by the systems and methods disclosed herein, looked.
The first camera 111 may be configured to determine a position of the vehicle with respect to a reference point. In some examples, the first camera may comprise a 2D camera. The first camera 111 may be located above the vehicle. For example, the first camera 111 may be located at a position above a roof of the vehicle.
The second camera 112 may be configured to scan the interior of the vehicle. In some examples, the second camera 112 may be configured to detect a seat position, a steering while position, an object present in the vehicle, and/or any combination thereof. The second camera 112 may be positioned outside the vehicle. The second camera 112 may be coupled to the robotic arm 110. In some examples, the second camera 112 may comprise a 3D camera.
In some examples, the first and second controller 113, 115 may be configured to operate as a single controller. In other examples, the first and second controllers 113, 115 may operate as separate controllers. The first and second controller 113, 115 may be configured to control the execution of tool paths and coordinate the dissemination of tools and motion paths to each of the robotic manipulator controllers and peripheral hardware. The first controller 113 may be also configured to transmit the instructions to the robotic arm 110 to execute, based on the scanned interior of the vehicle by the second camera 112, the cleaning operation in accordance with the created tool path. In some examples, the tool path represents not only the location on the surface that a path is being applied for motion execution, but also contains information such as metadata that includes the process information, such as stand-off, tool angle or an angle range, and tool. This information may be represented in a MSDP tool file schema. In some examples, the scheme includes a format for the tool paths and requirements for the specific tool, which may be assigned to any candidate surfaces within the cleaning domain that is the interior of the vehicle. The second controller 115 may be configured to receive data from one or more sensors to mitigate collision.
The communication system 114 may be coupled to any number of the controllers 113, 115. In some examples, the communication system 114 may include a quality assurance feedback loop, as previously described above with respect to quality assurance feedback 108.
The database 102 may include a plurality of stored vehicle configurations. In some examples, the database 102 may contain relevant interior and exterior vehicle data and characteristics which can be constructed into a 3D map of a vehicle's interior. Data in the database 102 may be organized in a way which allows the data to be attributed to and retrieved by a vehicle's make, year, and model number. Maps can also reflect the 3D map of the interior as designed, as well as observed variations caused by objects, added features or items, or from other modifications to the vehicle. As explained above, the system 100 may be configured to generate or retrieve existing 3D maps of vehicle interiors for instructing and controlling the robotic arm 110 of robot system. The controllers 113, 115 may be configured to compare map to real-time sensor-collected data and, in the event of differences, make decisions about the appropriate path to take based on a set of rules and priorities. The controllers 113, 115 may be configured to document and update 3D maps and instructions in the robotic arm 110 of the robot system and the database 102 with the new data.
The system 100 has two primary episodes of robot control/guidance: First, when the vehicle is presented on the conveyor 117 to the robotic arm 110 of the robot system—for guiding the robotic arm-pair 110 inside the vehicle through the door opening. Second, once inside the vehicle, the controllers 113, 115 may be configured to manage the motion, movements, and workflow of the robotic arm 110 and end effectors to complete the cleaning cycle in a timely manner.
The system 100 may be configured to continuously update the database 102 and 3D vehicle maps based on measured efficacy of motions of the robotic arms 110, using algorithms to ‘learn’ and optimize/minimize the motion path and time required to enter the vehicle and conduct the work. The learning and optimization can be attributed to, without limitation, specific vehicle makes, years, and/or models; types of vehicles, such as four-door sedans, trucks, or the like; and/or to commonly observed alterations/obstructions within a vehicle interior (e.g., how to best navigate around a baby seat).
In some examples, a computer system may be configured to maintain the database 102, and physically or wirelessly connect to various input, output, and backup components of the system 100. The system 100 may be configured to provide data processing or computational support as needed. The system 100 can be connected to a network, such as the network described above, such that information stored therein can be accessed via the network at multiple locations, including locations along the same vehicle wash line, such as downstream of the scanning system, and including remote locations, such as at a vehicle wash located in different locations, including a different city and/or state.
The rail 116, such as a linear rail, may be located above and alongside the vehicle. In some examples, the robotic arm 110 may be configured to move along the linear rail 116.
The vehicle may be carried on a conveyor 117. In some examples, the robotic arm 110 may be configured to move along the linear rail 116 in coordination with a motion of the conveyor 117.
The state machine 118 may be configured to manage timing and coordination of a plurality of cleaning implements and system peripherals. The cleaning method and system disclosed herein includes two workflows: actions at an individual robot station, and steps required to fully process a vehicle as it transits through the various cleaning stages. In some examples, multiples of these workflows may be happening in parallel: for example, six robotic arms 110 and three vehicles at a time. The state machine 118 may be configured to track and direct sequencing of these parallel tasks. Each vehicle in the system 100 may be associated with a “vehicle task list” that is configured to define which tasks are required to fully process a vehicle. Without limitation, the tasks may include computation (motion planning), sensing (overhead location scan), robot motion (execute front driver door cleaning paths). The tasks may be executed in parallel (motion planning of various sub-paths) or in a sequence (motion execution requires motion planning to be completed). The state machine 118 may be further configured to process active vehicle task lists (for each in-process vehicle) to run each task on asynchronous background threads when the appropriate preceding tasks have been completed.
In some examples, the system 100 may include a safety system. For example, the safety system may be configured to utilize vision methodologies to sense when humans or other undesired obstacles are close to the interconnected robot system or end effectors of the robotic arm 110, resulting in a reduction in robot system force or halting robot activity. The system 100 may be designed to effectively accommodate feedback, measurement, and control specific to robotic arms 110. For example, the robotic arms 110 may be used which are designed with greater mass and payload, and operate at higher speeds than collaborative robots for a lower investment and improved operational efficiency. In some examples, barriers, such as hard barriers, may be placed to ensure non-workers do not approach or get near working interconnected robot systems. Barriers can be transparent curtain-wall like structures made of glass, acrylic, or polycarbonate, or fence-like structures, or walls.
Any of the components of the system 100 may be implemented as hardware and/or software. Without limitation, any of the components of the system 100 may include a processor and a memory that communicate with each other through one or more networks that may also be part of the systems 100 of
The processor may comprise an application specific integrated circuit and may be configured to execute one or more instructions. The processor may be part of a device, such as a network-enabled computer. As referred to herein, a network-enabled computer may include, but is not limited to a computer device, or communications device including, e.g., a server, a network appliance, a personal computer, a workstation, a phone, a handheld PC, a personal digital assistant, a thin client, a fat client, an Internet browser, or other device. The device also may be a mobile device; for example, a mobile device may include an iPhone, iPod, iPad from Apple or any other mobile device running Apple's iOS operating system, any device running Microsoft's Windows Mobile operating system, any device running Google's Android operating system, and/or any other smartphone, tablet, or like wearable mobile device. In some examples, the server may be configured as a central system, server or platform to control and call various data at different times to execute a plurality of workflow actions to perform one or more functions described herein. The one or more servers may contain, or be in data communication with, one or more databases.
The device can include a processor and a memory, and it is understood that the processing circuitry may contain additional components, including processors, memories, error and parity/CRC checkers, data encoders, anticollision algorithms, controllers, command decoders, security primitives and tamper proofing hardware, as necessary or desired to perform the functions described herein. The device may further include a display and input devices. The display may be any type of device for presenting visual information such as a computer monitor, a flat panel display, and a mobile device screen, including liquid crystal displays, light-emitting diode displays, plasma panels, and cathode ray tube displays. The input devices may include any device for entering information into the user's device that is available and supported by the user's device, such as a touchscreen, keyboard, mouse, cursor-control device, touchscreen, microphone, digital camera, video recorder or camcorder. These devices may be used to enter information and interact with the software and other devices described herein.
The memory may be a read-only memory, write-once read-multiple memory or read/write memory, e.g., RAM, ROM, and EEPROM, and the system of
In some examples, the network may be one or more of a wireless network, a wired network, or any combination of wireless network and wired network, and may be configured to connect to any components of system. For example, the network may include one or more of a fiber optics network, a passive optical network, a cable network, an Internet network, a satellite network, a wireless local area network (LAN), a Global System for Mobile Communication, a Personal Communication Service, a Personal Area Network, Wireless Application Protocol, Multimedia Messaging Service, Enhanced Messaging Service, Short Message Service, Time Division Multiplexing based systems, Code Division Multiple Access based systems, D-AMPS, Wi-Fi, Fixed Wireless Data, IEEE 802.11b, 802.15.1, 802.11n and 802.11g, Bluetooth, NFC, Radio Frequency Identification (RFID), Wi-Fi, and/or the like.
In addition, the network may include, without limitation, telephone lines, fiber optics, IEEE Ethernet 902.3, a wide area network, a wireless personal area network, a LAN, or a global network such as the Internet. In addition, the network may support an Internet network, a wireless communication network, a cellular network, or the like, or any combination thereof. The network may further include one network, or any number of the exemplary types of networks mentioned above, operating as a stand-alone network or in cooperation with each other. The network may utilize one or more protocols of one or more network elements to which they are communicatively coupled. The network may translate to or from other protocols to one or more protocols of network devices. Although a network may be depicted as a single network, it should be appreciated that according to one or more examples, the network may comprise a plurality of interconnected networks, such as, for example without limitation, the Internet, a service provider's network, a cable television network, corporate networks, and home networks.
For highly translucent or transparent surfaces (glass), a scanning spray is used. Without limitation, the scanning spray may include AESUB 3D™ scanning spray. It is understood that other scanning sprays may be used and that the scanning spray is not limited to such a scanning spray. The scanning spray may be applied, for example automatically or manually, to an exterior of the window while scanning the interior of the window to obtain accurate curvature so as to avoid unevenness of the applied spray. Regarding the application amount of the scanning spray, a light coating may be sufficient, and the scanning spray may evaporate on its own without requiring any cleaning or removal. In some examples, blue painter tape may be applied on interior surfaces of the vehicle that are difficult to reach via the scanning spray.
When scanning one or more doors or one or more portions thereof, it is preferred that the method includes scanning from the outside, which prevents surfaces sticking through each other if any door or portion thereof is moved slightly, e.g., a bump of the door interior may be done first, then it may protrude through scan of the exterior due to the swing and movement of the door. The one or more doors of the vehicle may be scanned separately from the frame and interior. This is to manage the situation where doors are bumped during scanning or move slightly. Overlapping geometry may be deleted after scanning, this avoids undesired overlaps when merging various scans.
Additional exterior data may be gathered for collision avoidance geometry. For instance, for regions of the vehicle that are not be subjected to robot tool path planning, or obstacle identification, or other discrete elements that influence robot motion planning within the vehicle, the general shape of the vehicle is captured for collision avoidance. This includes the exterior of the vehicle from A pillar to the front, and C pillar to the rear. For this process, a patterned blanket may be clipped to the vehicle and the scanning traverse speed by the operator was observed to be faster without repeating seen surfaces to perform detail image filling/completion. The software then takes the point cloud data that matches/aligns images due to the texture tracking of the blanket to create a uniform mesh surface that is sufficiently accurate to enable motion planning software to avoid those surfaces to prevent any sort of collision during motion execution. Models may be generated and merged from the various master scans of the various regions to form a single model, and this model may be decimated to reduce total size clean-up processes, which may be manual in some examples and may be automated in other examples, may be configured to eliminate holes, fill in regions with no data, in which it is undesirable for any robot to enter (for example, such as under one or more seats of the vehicle), and fix any overlapping as noted above. Thus, adding textured surfaces over portions of the vehicle that are only needed for collision avoidance may help accelerate scanning and processing and improve overall process flow. Moreover, partial car covers having markers printed, secured, or otherwise coupled thereon can be used.
Once a completed model is captured, segmentation may be initiated. This is the process of segmenting out portions of the scanned data that identify one or more features of interest. These include but are not limited to one or more seat bottoms, floor wells, and dash/console areas. The segmentation may be configured to drive a specific process type, such as the spray of a cleaning agent, or tool paths for a vacuum nozzle. In some examples, a trained classifier may or may not be included. While in some examples, a priori knowledge may be present, the MSDP has no a priori knowledge (for example, the MSDP may be yield the presence of a seat bottom, a dashboard, or the like.
During tool path planning, tool paths process steps may be configured to plan desired tool paths on as-presented master scans based on application of one or more rules entered by the user for the specific process. The one or more rules may include, but are not limited to, tool offset from the target surface, work and travel angle, as well as raster spacing. Though a mesh model may be of sufficient quality, a primitive can be inserted over the surface, such as at a predetermined z-offset distance, to provide a more consistent motion path for a subsequent motion plan. For example, if the mesh model includes highly tufted seats, a tool path motion using the mesh data could be overly complex, which adds unnecessary time to the process and may not lead to any improvements in cleaning performance. Thus the use of a primitive or sets of primitives may provide for more simple motion profiles to be planned and executed, thereby increasing computations performance and motion consistency. In some examples, auto-fit of a primitive just above the target surface may be human-guided. In other examples, the auto-fit of a primitive just above the target surface may be automated. A region is initially segmented, once surfaces are classified/identified, the primitive is fit, e.g., a plane that may be a 2D polygon, then subsequent raster paths are planned on the plane as opposed to directly onto the mesh. The output mesh from the scan, the identified regions that are segmented, collision geometry, and the tool paths for the target processes may constitute the MSDP. In some examples, tool paths may be applied directly to filtered mesh surfaces and additional filtering via leverage of a cartesian motion planner to provide refined tool paths based on the actual mesh surface may be leveraged, so as to reduce reliance on primitive fitting and allowing for improved tool following in execution.
Understanding when a master scan data package is required is used in implementing the systems and methods disclosed herein. For example, it is reasonable to say that if there is an MSDP, for a 2020 Toyota Camry, one may not need to be created for a 2021 Toyota Camry. This concept may be referred to as “generalizability” and is used to define how much of the data creation occurs over the various types of vehicles that may enter the vehicle wash operation. In general, it is believed that MSDPs may be shared across a subset of a type of vehicle, such as noted across the Camry line for at least a finite set of model years. Ideally, and as the system 100 progresses in capability, a broader range, such as full sized 4 door sedans, may be handled that takes into account managing variations such as seat width, similar to how obstructions would be managed by simply clipping tool planned tool paths. In some examples, the aforementioned generalizability may take into account notations on executed tool paths on the line to help inform whether any number of variations of an MSDP for a future vehicle in the same line (continuing with the above example, a 2025 Camry), is needed.
The MSDP may include detailed surface models, segmented out regions of the interior, collision modeling of surfaces, and specific tool paths on the segmented surfaces that are relevant to the desired process for execution. In some examples, segmentation may be human assisted, e.g., a human may input information into the client device, such as highlight an area (such as surfaces, dashboards, or seat bottoms) via the application comprising instructions for execution on the client device which may identify it as a given region, such as a dashboard. In other examples, these regions may be autonomously identified either via a rules-based algorithm, an AI-assisted semantic segmentation algorithm, or any combination thereof that is configured and optimized for automotive interior surfaces. Once segmentation has taken place, tool path planning adhering to the rules for the specific process may take place. Once tool paths are approved, or a quality assurance which leverages feedback from end of line operators, they may be uploaded to the MSDP database, which may be a cloud service to enable MSDP information be leveraged by any site that has access to the data. Having data in the cloud and collecting information from sites on how MSDP is both generalizable, and how it performs enables richer capability for optimization via learning, which may further leverage the cloud infrastructure that is leveraged, further reducing the amount of master scanning that has to take place over time and thereby mitigating the need for storage and improving overall system efficiency.
Operations may be configured to run “on the line” even at first deployment. The cycle management coordinator may include a state machine 118 that is configured to handle the timing and coordination of developed sub-modules and interaction with components of the system 100. This may include the launch of configuration scan and alignment, receiving the updated information, and providing the updated tool path offsets to the motion planner. Specific motion plans are assigned to specific assets and then, in a coordinated fashion, dispatched to the specific robot on the line, utilizing the specific hardware with the specific process recipes.
The configuration scan and alignment is the process that the vehicle to be processed is located on the line and is matched to the target or selected MSDP in the database. Information may be acquired about the vehicle that is entering a bay of a car wash, and this acquisition may be obtained through existing car wash infrastructure. The appropriate MSDP is then acquired, and the configuration scan and alignment process may be initiated. An overhead camera, such as the overhead 2D camera of
Regarding step 510, a method using an iterative closest point (ICP) algorithm is implemented and configured in order to align the car body master scan to a configuration scan that includes the geometry of the front doorjamb. The door jamb area was selected because it provides consistent geometric features that are helpful for registration. This registration result provides the location of the vehicle body and seat relative to the configuration scan. After that, a subsequent registration is performed to locate the seat position more precisely inside the vehicle. In some examples, the seat master scan may include two separate mesh models, such as, the bottom and backrest, aligned independent of each other. After aligning the car body and seat, the algorithm may be configured to remove the geometry that is common between the master and configuration scans and labels what remains as an obstacle. This creates a tiered ICP algorithm implementation that refines detailed alignment after first bulk alignment. For example, an ICP algorithm implementation, such as in MeshLab, may be configured to select any number of common points between the master and configuration scans, in which the ICP algorithm may be configured to then make any adjustments for final alignment. It is understood that other algorithms may be configured in lieu of MeshLab.
By aligning a master scan to the configuration scans, an application can be created that allows for determination of the vehicle location. In addition, seat positions may be accurately located and obstacles may be identified that keep the robot from reaching certain areas.
Without limitation, the obstacles may include an umbrella, a device mount (such as a phone mount), a drink cup, a jacket, an infant car seat. During testing, black, metal coffee mugs were often barely visible using a 3D camera (such as a Zivid™ camera), whereas a depth camera (such as a RealSense D455™ camera) provided better recognition of the black, metal coffee mug. It is understood that other types of suitable cameras may be implemented. In addition, to make up for the smaller field of view, two configuration scans can be combined when using the 3D camera.
To simulate a production system condition, an offset may be applied to the configuration scans. For example, a 0.2 radian (˜12 degree) and 0.1 m offset may be applied to configurations scans.
Initially, a view of the door jamb of the vehicle may be collected. This may comprise collecting views of both front and rear doorjambs. Then, a plurality of views of the interior through the front door are collected. Without limitation, this may include collecting three views. For each of the plurality of views, the following process may take place: collecting a configuration scan with no obstacle; adjusting the seat and/or adding an obstacle; and collecting the configuration scan. If the seat is adjustable, a plurality of seat positions, such as three seat positions, may be collected. After the views of the interior through the front door are collected, a plurality of views of the interior through the rear door may be collected. As with the interior through the front door view collection, this may include collecting three views of the interior through the rear door. For each of the plurality of views, the following process may take place: collecting a configuration scan with no obstacle; adjusting the seat and/or adding an obstacle; and collecting the configuration scan. If the seat is adjustable, a plurality of seat positions, such as three seat positions, may be collected.
In some examples, inputs for the following may include: for master scans, vehicle body without seat and steering wheel, seat bottom, and seat backrest; for configurations scans, front doorjamb, front interior (several views captured at various seat positions and with obstacles), rear interior (several views captured with different obstacles), and rear door jamb (optional); for configuration parameters, alignment constraints and obstacle detection. Outputs included: location of the vehicle body; location of the seat bottom and backrest; obstacle 3D point cloud; registration metric including a match percentage (which may measure number of common points between the master and configuration scans).
Prior to sending instructions to the robot for execution of one or more cleaning operations, a process plan is created. From the prior operation, the system may have tool paths aligned to the frame of the robot relative to the vehicle. However, motions that are coordinated with the motion of the vehicle on a conveyor, for example, are planned as well as approaches and departures for the different processes, any tool changes and the transitions from free space (joint space) motion to cartesian (tool path) motion. In some examples, this process planning pipeline for the management of cleaning tasks inside vehicles may comprise: sequencing of robot planning tasks for the completion of a series of cleaning toolpaths (For example: Using planning frameworks; defining planning instructions using YAML files, this is a human readable configuration format that may be updated via a user interface, to allow tuning by technicians; sparse planning of cartesian trajectories; implementing an algorithm, such as dijkstra's shortest path graph search algorithm, to plan robot trajectories for cartesian paths using sampler based approach to more quickly find the configuration for the various poses along the cartesian path and edge evaluators. In some examples, the edge evaluator may account for tool speed limits, this is used by software tools to determine if a move between two consecutive points does not exceed the maximum allowed tool speed. The planning instructions program, written in YAML file format, may be configured to define sequence of free-space and cartesian toolpath planning instructions, thus supporting two kinds of planning instructions. The cartesian toolpath instructions may be configured to specify properties, such as tool standoff, approach and retreat distance, transformations, and coordinate frame of reference. For example, each instruction may be configured to specify a profile for each step in the planning process. Tool path planners may be configured to facilitate optimal configurations of the manipulators in the motion planning of cartesian tool paths. In some examples, other tool path planners may be used. Adding the ability to enable technician configuration enables this to be optimized for the specific application. From this point, the output of the software tools may be fed as a seed into an optimization pipeline to refine the manipulator free space motion based on defined constraints. These may be defined by acceleration or torque limitations defined by the system 100, or may also be adjusted based on technician and end-user end of line feedback on performance. Further performance enhancements may or may not be realized by the inclusion of a reinforcement learning actor as a seed for the optimization-based motion planning.
The systems and methods described herein employ process planning 106. Process paths for various processes of interest (including but not limited to applying a solvent or spray or wiping on any number of vehicle interior surfaces) may have a unique transition to departure, that is coordinated with other paths for completion within a given time frame. For example, the process paths may be configured to be completed within 80 seconds, which marks a departure from domains which are not time-constrained. In some examples, the systems and methods described herein may be implemented in an existing car wash workflow, where a vehicle may be configured for conveyance along a belt in a car wash bay. Additionally, or alternatively, the systems and methods are applicable to stationary vehicles in which the various process steps can be moved into place and conducted in sequence. In other words, the interior of the vehicle may be cleaned according to the systems and methods described herein while the vehicle remains still, either on or off the conveyor belt. Further still, the systems and methods can be employed in a car wash workflow using a combination of moving and unmoving stations. The systems and methods described herein may be generalizable in that there exists the ability to train a model and perform one or more process paths with limited data that is collected. The systems and methods are configured to adapt and modify execution of these process paths based on the limited data acquired, which can avoid the need to create a master scan data package for each car configuration.
The motion planning algorithm, in order to plan a new trajectory, may be configured to utilize toolpath processing; cartesian planning; optimization-based free-space planning; and post-processing.
In tool path processing, changes to the original tool path may be applied prior to planning. For example, this may include up-sampling or down-sampling waypoint count, etc. In Cartesian planning, a planner may be configured to obtain valid robot positions for toolpath points as allowed by the specified tool constraints. In free-space planning, this may be used for planning a free-space move from a most recent robot position to a start of the cartesian robot trajectory. Under post-processing, time values may be computed for trajectory waypoints so as to move the tool at a desired speed while at the same time adhering to the speed limits of the robot.
The steps above can use its own set of parameters, which can be specified in “profiles.” These profiles allow to easily load a different set of parameters in order to test how the results change.
A robot trajectory executor may be configured to execute robot free space and car cleaning tool path trajectories in the proper order, monitor the vehicle position, via a node (instantiated software code that performs a specific function) that tracks via external measuring or belt encoder monitoring, or a combination of both, to actively command a robot linear rail and the manipulator when the car is within the working envelope (reach) of the robot; and send back the robots to the start/initial position when the robot has completed the provided trajectories.
The sequence of planned robot trajectories may be sent to the visualization on the user interface of the client device for viewing, along with the option to halt or approve and send to the industrial robot controller for execution. In some examples, the display of the user interface of the client device may be configured to allow for further inspection of the trajectories, and the instructions for execution may be aborted or may be re-planned after operator intervention. The solution enables for the storage of successfully executed trajectories, which enables further optimization as the solution matures, up to including a reinforcement learning implementation for the optimization of industrial application trajectories. Thus, the database includes successful trajectories and the specific data set that informs the reinforcement learning implementation.
Once the process plans and motion plans are complete, including coordination of external axes that are complete and either simulated and approved, or approved via one or more algorithms, these motion plans are converted to native industrial robot controller software program language and transferred to the specific robot controllers for execution. A programmable logic controller (PLC) may be configured to coordinate the call of the specific programs for execution and act as a master of the hardware components of the system. A device, including but not limited to an industrialized personal computer, that contains the developed software may be configured to interact with the PLC to facilitate the coordination of the hardware assets. Once the desired processes are executed, the vehicle may depart the robotic operational area and thereby complete the process. In some examples, an operator can review the work performed and input via a touch screen of a device, including but not limited to a mobile device, information indicative of whether any rework was performed or assess if regions were skipped that possibly should not have been skipped. A skip may refer to a scenario where there was something within the vehicle that did not align between configuration data and master data. This results in collision geometry and may then remove the impacted/obstructed tool paths, when then results in the region not being processed. This data may be utilized for continuous improvement by adding annotations to the MSDP.
At block 2010, the method 2000 may include determining a position of the vehicle with respect to one or more robotic arms that are positioned exterior to the vehicle. At block 2020, the method 2000 may include scanning a configuration of the vehicle. For example, the method 2000 step may yield a configuration scan. In some examples, scanning the configuration of the vehicle may include capturing a plurality of images of the vehicle. For example, the configuration scan may capture exterior images and interior images of the vehicle.
In some examples, prior to scanning a configuration of the vehicle and block 2020, the method 2000 may also include acquiring information about the vehicle by retrieving master data from a database. The method 2000 may also include aligning data from the configuration scan with the master data. The method 2000 may also include detecting obstacles based on discrepancies between the configuration scan and the aligned master data. In some examples, the plurality of tool paths may be created to avoid the detected obstacles by a predetermined distance.
At block 2030, the method 2000 may include identifying any number of surfaces to be cleaned. In some examples, these surfaces may include interior vehicle surfaces. At block 2040, the method 2000 may include creating a plurality tool paths in MSDP, in which robot trajectories are generated for execution of the MSDP-created tool paths for the one or more robotic arms to clean the identified surfaces. The tool paths may be created as part of the master data package. A tool path(s) may be applied to meshes with appropriate tool information, such as stand-off allowable tool and travel angles. This information may be included in the data scheme that is associated with each segmented region/component. In some examples, creating the plurality of tool paths may include implementing an algorithm that is configured to plan trajectories for the one or more robotic arms using a sampler-based approach. In some examples, the method 2000 may include creating optimized tool paths by running the created plurality of tool paths through a free space motion system and/or cartesian motion planning system.
At block 2050, the method 2000 may include controlling the one or more robotic arms to move along the created tool paths and execute a cleaning operation in the interior of the vehicle. In some examples, the one or more robotic arms may move along the motion planned tool paths to execute a cleaning operation relative to, including but not limited to, the identified surfaces. The tool paths may be clipped/truncated and take into account, for example, steering wheel position, etc. in motion planning or robot trajectory planning. Once the trajectories are planned, they may be sent for execution in association with the cleaning operation.
The method 2000 may further include optimizing a sequencing of a plurality of tasks relative to a time permitted for each task of the plurality of tasks.
At block 2110, the method 2100 may include storing a plurality of images for a plurality of vehicles in a database to create a master data package. In some examples, and without limitation, the master data package may include a vehicle make and model data, vehicle year data, vehicle class data, and/or any combination thereof.
At block 2120, the method 2100 may include acquiring vehicle specific data from the master data package. At block 2130, the method 2100 may include scanning a configuration of the vehicle. At block 2140, the method 2100 may include aligning the acquired vehicle specific data with the scanned configuration. At block 2150, the method 2100 may include creating a process plan for execution of a cleaning operation. At block 2160, the method 2100 may include sending instructions to a robot to execute a plurality of cleaning operations based on the created process plan. At block 2170, the method 2100 may include executing the cleaning operations according to the instructions.
In some examples, a cycle time, which may be between a start of the scanning the configuration of the vehicle and an end of the execution of the plurality of cleaning operations, may be less than a predetermined time duration. For example, the cycle time may be less than five minutes. In some examples, the method 2100 may also include providing feedback on a quality of the executed plurality of cleaning operations.
Each pair of robotic arms 110 may be configured to employ a sliding cantilever structure 2310 allowing the arm-pair to be positioned horizontal to the floor or perpendicular to it, or controlled to any position in-between. Each cantilever structure 2310 is fastened to and supports a pair of robotic arms 110, and is placed next to the vehicle on either side, with two cantilever structures 2310 per side of the vehicle, next to a front and rear opening for a total of four robotic arm 110 pairs. The cantilever structures 2310 are attached to vertical tracks that allow the robotic arms 110 to rapidly move up and down to reach the optimal height for entering the vehicle and reaching interior surfaces of the vehicle interiors through both the front and back entry doors simultaneously. The cantilever structures 2310 are fastened to a deck on either side of the conveyor 117.
In another example, the system 100 may employ a fifth and/or sixth cantilever structure 2310, each with a robotic arm pair 110, placed toward the rear of the vehicle on a linearly actuated movable platform, so that it can clean the rear portion of the vehicle, trunk and/or hatch, at the same time as the primary interior cleaning operations.
The interconnected robot system allows for rapid actuation and end effector placement of the robotic arms 110. The controllers 113, 115 may be configured for high-speed pick and place operations where speed and accuracy are paramount to success of cleaning the interior vehicle by system 100. Current processing speeds allow image-capture-to-robot-arm-movements in as low as 500 milliseconds and control 1,400 actuated work steps per hour up to 1 meter apart.
Each pair of robotic arms 100 are joined in a horizontal position, with the two robotic arms 110 reaching in the same direction, with one lying horizontally atop the other. The robotic arms 110 may work in tandem, each armed with a single- or double-tool end of arm effector. Duties for each robotic arm 110 can be programmed by the interconnected robot system and typically assign one robotic arm 110 to focus on upper portions of the vehicle interior and the other robotic arm 110 the lower portions of the vehicle interior. In some examples, one robotic arm 110 may address window cleaning, followed by cleaning interior door panels, drying the door frame and cleaning the jamb—while the other robotic arm 100 may simultaneously focus on upholstery and floor cleaning. The tooling has been designed so that multiple cleaning effectors are mounted on the robotic arm 110 at once so that the end effectors do not need to be physically removed from the robotic arm 110 to go on to the next task but, instead, can be rotated into and out of place while attached to the robotic arm 110. Coordination and managing collision avoidance between the robotic arms 110, end effectors, and elements of the vehicle may be conducted by the interconnected robot system.
Each interconnected robot system can work in parallel to the others, so that each system and the associated robotic arms 110 are working simultaneously to clean approximately one-quarter of the vehicle interior.
The end effectors of the robotic arms 110 may be configured specifically for the primary robotic vehicle interior cleaning tasks and their throughput requirements. As the robotic arms 100 and end effectors carry out their functions, they may be configured to detect and collect data through a variety of sensors, including: motion, force, vision, torque, and/or other sensors attached to them. The purpose of this data is not only to determine cleaning efficacy but to also refine the 3D maps of the interior for path planning optimization. These data are fed to the path planning process which controls the movements of the robotic arms 110 and tools. In some examples, the tasking of tools may be first handled in the MSDP and tool path creation process, where a tool and its requirements are assigned to specific paths. The tasking and coordination at runtime is handled by a state machine, such as state machine 118 as previously explained above. Over time, the aggregation of data from multiple vehicles will allow the software and its algorithms to determine optimal motion pathways and enable faster responses to non-standard items within the vehicle (such as baby seats, aftermarket items, etc.), further reducing cycle times and speeding the process of cleaning vehicle interiors.
By way of example, the workflow of the systems and methods described herein has been designed, configured, and optimized to minimize cost by retaining human activities least cost-effective for automation—such as placement of vehicle on the conveyor, opening doors, removing and replacing removable items out of and into the vehicle, handling one-off items, and conducting final quality control. In addition, human workers may be responsible for cleaning rear- and forward-facing window glass interior surfaces, cleaning the trunk area, and removing, cleaning, and replacing floor mats. In this regard, some zones include human workers present on the conveyor system whereas other zones include robot system and are designed to prevent human presence to reduce the risk of injury. The systems and methods described herein focus the robotic activities on those prone to poor quality, cumbersome to complete, and/or those posing the most risk to worker comfort, safety, and vehicle damage.
Throughout the specification and the claims, the following terms take at least the meanings explicitly associated herein, unless the context clearly dictates otherwise. The term “or” is intended to mean an inclusive “or.” Further, the terms “a,” “an,” and “the” are intended to mean one or more unless specified otherwise or clear from the context to be directed to a singular form.
In this description, numerous specific details have been set forth. It is to be understood, however, that implementations of the disclosed technology may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description. References to “some examples,” “other examples,” “one example,” “an example,” “various examples,” “one embodiment,” “an embodiment,” “some embodiments,” “example embodiment,” “various embodiments,” “one implementation,” “an implementation,” “example implementation,” “various implementations,” “some implementations,” etc., indicate that the implementation(s) of the disclosed technology so described may include a particular feature, structure, or characteristic, but not every implementation necessarily includes the particular feature, structure, or characteristic. Further, repeated use of the phrases “in one example,” “in one embodiment,” or “in one implementation” does not necessarily refer to the same example, embodiment, or implementation, although it may.
As used herein, unless otherwise specified the use of the ordinal adjectives “first,” “second,” “third,” etc., to describe a common object, merely indicate that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
While certain implementations of the disclosed technology have been described in connection with what is presently considered to be the most practical and various implementations, it is to be understood that the disclosed technology is not to be limited to the disclosed implementations, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
This written description uses examples to disclose certain implementations of the disclosed technology, including the best mode, and also to enable any person skilled in the art to practice certain implementations of the disclosed technology, including making and using any devices or systems and performing any incorporated methods. The patentable scope of certain implementations of the disclosed technology is defined in the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims.
This application claims priority to U.S. Provisional Patent Application No. 63,256,763 filed on Oct. 18, 2021, the contents of which are incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63256763 | Oct 2021 | US |
