Patent Application
20230339734
The disclosure generally relates to an object detection system and method on a work machine.
Work machines are configured to perform a wide variety of tasks for use as construction machines, forestry machines, lawn maintenance machines, as well as on-road machines such as those used to plow snow, spread salt, or machines with towing capability. Accordingly, different attachments may be coupled to the work machine such as a bucket, rotary attachments, plows, spreaders, and transport. The work machines are therefore equipped with one or more interfaces to which different attachments may be coupled. Such interfaces may include a hitch in the rear of the work machine, or a Quick-Tach coupler in the forefront of the work machine, for example. When coupling an attachment to a work machine, an object detection system coupled to the work machine may result in a false positive and therefore disrupt the flow of function. Therein lies an opportunity to improve function for a more efficient operation.
An object detection system and method therefore are disclosed. The object detection system comprises of a frame, a boom arm coupled to the frame, an image sensor, a processor, and a controller. The image sensor is coupled to one of the boom arm and the frame for capturing an image. The processor is communicatively coupled to the image sensor and recognizes an object in the image. The controller is configured to execute a function of the work machine when the object is recognized; and override execution of the function of the work machine when the object is defined as the target object.
The system may further comprise a display device displaying an icon representing the recognized object on a display device. Defining the recognized object as the target object may include manually selecting the icon displayed on the display device. Alternatively, defining the recognized object as the target object includes the controller automatically defining the recognized object as the target object based on one of identification as a pre-defined object stored in a memory, or identification as a previously defined target object.
Additionally, the object stored in memory may be received from one of a second work machine, a worksite control center, and a predefined program.
Recognition of the object occurs within a defined space relative to the work machine, wherein the defined space being up to a predefined distance from the work machine.
Additionally, a target object may be untargeted if the target object falls outside a field of view on the display device.
A function of the work machine may comprise one of alerting an operator, stopping the work machine, modifying a current travel speed of the work machine, and steering the work machine.
The method of controlling a work machine having an object detection system includes capturing an image with an image sensor, recognizing an object in the image with the object detection system, defining the recognized object as a target object, operating the work machine and overriding a function of the object detection system when the object is defined as the target object. The target object may further become untargeted if the target object falls outside a field of view on the display device. The function of the object detection system may comprise one or more object avoidance and object engagement. A function of the object detection system may include of alerting an operator, stopping the work machine, modifying a current travel speed of the work machine, and steering the work machine.
The above features and advantages and other features and advantages of the present teachings are readily apparent from the following detailed description of the best modes for carrying out the teachings when taken in connection with the accompanying drawings.
Those having ordinary skill in the art will recognize that terms such as “above,” “below,” “upward,” “downward,” “top,” “bottom,” etc., are used descriptively for the figures, and do not represent limitations on the scope of the disclosure, as defined by the appended claims. Furthermore, the teachings may be described herein in terms of functional and/or logical block components and/or various processing steps. It should be realized that such block components may be comprised of any number of hardware, software, and/or firmware components configured to perform the specified functions.
Terms of degree, such as “generally”, “substantially” or “approximately” are understood by those of ordinary skill to refer to reasonable ranges outside of a given value or orientation, for example, general tolerances or positional relationships associated with manufacturing, assembly, and use of the described embodiments.
As used herein, unless otherwise limited or modified, lists with elements that are separated by conjunctive terms (e.g. “and”) and that are also preceded by the phrase “one or more of” or “at least one of” indicate configurations or arrangements that potentially include individual elements of the list, or any combination thereof. For example, “at least one of A, B, and C” or “one or more of A, B, and C” indicates the possibilities of only A, only B, only C, or any combination of two or more of A, B, and C (e.g., A and B; B and C; A and C; or A, B, and C).
As used herein, “controller” 66 is intended to be used consistent with how the term is used by a person of skill in the art, and refers to a computing component with processing, memory, and communication capabilities, which is utilized to execute instructions (i.e., stored on the memory 90 or received via the communication capabilities) to control or communicate with one or more other components. In certain embodiments, the controller 66 may be configured to receive input signals in various formats (e.g., hydraulic signals, voltage signals, current signals, CAN messages, optical signals, radio signals), and to output command or communication signals in various formats (e.g., hydraulic signals, voltage signals, current signals, CAN messages, optical signals, radio signals).
The controller 66 may be in communication with other components on the work machine 100, such as hydraulic components, electrical components, and operator inputs within an operator station of an associated work machine. The controller 66 may be electrically connected to these other components by a wiring harness such that messages, commands, and electrical power may be transmitted between the controller 66 and the other components. Although the controller 66 is referenced in the singular, in alternative embodiments the configuration and functionality described herein can be split across multiple devices using techniques known to a person of ordinary skill in the art.
The controller 66 may be embodied as one or multiple digital computers or host machines each having one or more processors, read only memory (ROM), random access memory (RAM), electrically-programmable read only memory (EPROM), optical drives, magnetic drives, etc., a high-speed clock, analog-to-digital (A/D) circuitry, digital-to-analog (D/A) circuitry, and any required input/output (I/O) circuitry, I/O devices, and communication interfaces, as well as signal conditioning and buffer electronics.
The computer-readable memory 90 may include any non-transitory/tangible medium which participates in providing data or computer-readable instructions. The memory 90 may be non-volatile or volatile. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Example volatile media may include dynamic random-access memory (DRAM), which may constitute a main memory. Other examples of embodiments for memory 90 include a floppy, flexible disk, or hard disk, magnetic tape or other magnetic medium, a CD-ROM, DVD, and/or any other optical medium, as well as other possible memory devices such as flash memory.
The controller 66 includes the tangible, non-transitory memory 90 on which are recorded computer-executable instructions, including a monitoring algorithm 92. The processor 88 of the controller 66 is configured for executing the monitoring algorithm 92. The monitoring algorithm 92 implements a method of monitoring and/or detecting objects 85 near the work machine 100.
As such, a method 600 may be embodied as a program or algorithm operable on the controller 66. It should be appreciated that the controller 66 may include any device capable of analyzing data from various sensors, comparing data, making decisions, and executing the required tasks.
Referring now to the drawings,
A power source 165 is coupled to the frame 110 and is operable to move the work machine 100. The illustrated work machine 100 includes wheels, but other embodiments may include one or more tracks or wheels that engage the surface 135. In this exemplary embodiment, the ground-engaging mechanism 155 on the left side of the work machine 100 may be operated at a different speed, or in a different direction, from the ground-engaging mechanism 155 on the right side of the work machine 100. In a conventional skid steer, the operator can manipulate controls from inside an operator cab 160 to drive the wheels on the right or left side of the work machine 100 using a control device such as a joystick, a foot pedal, a touchscreen, and a steering wheel. The movement for work machine 100 may be referred to as roll 130 or the roll direction, pitch 140 or the pitch direction, and yaw 145 or the yaw direction.
The work machine 100 comprises the boom assembly 170 coupled to the frame 110. The attachment 105 (may also be referred to as work tool) may be coupled at a forward portion of the boom assembly 170 (e.g. a forklift) or alternatively in the rear portion of the frame 110 (e.g. a hitch 210), while the rear portion of the boom assembly 170 is pivotally coupled to the frame 110. The attachment 105 at the forward portion of the boom assembly 170 may be coupled through an attachment coupler 185, an industry standard configuration or a coupler universally applicable to many Deere attachments and several after-market attachments.
The boom assembly 170 of the exemplary embodiment, comprises a first pair of boom arms 190 (one each on a left side and a right side) pivotally coupled to the frame 110 and moveable relative to the frame 110 by a pair of boom hydraulic actuators (not shown), wherein the pair of boom hydraulic actuators, may also be conventionally referred to as a pair of lift cylinders (one coupled to each boom arm) for a skid steer. The attachment coupler 185 may be coupled to a forward section, or portion, of the pair of boom arms 190, being moveable relative to the frame 110 by a pair of tilt hydraulic cylinders (not shown). The frame 110 of the work machine 100 further comprises a hydraulic coupler (not shown) on the front-end portion 120 of the work machine 100 to couple one or more auxiliary hydraulic cylinders to drive movement of or actuate auxiliary functions of the attachment 105. The hydraulic coupler, contrary to the attachment coupler 185, enables the hydraulic coupling of the hydraulic actuators(s) on the attachment 105 to a hydraulic system of the work machine 100. Please note that not all attachments have one or more auxiliary hydraulic cylinders and therefore will not use the hydraulic coupler. Alternatively, uses for the hydraulic coupler add another form of movement such as lifting or lowering a forklift 205, opening or closing a grapple type attachment, spinning a rotary drum, or turning the cutting teeth on a trencher, to name a few. An image sensor 195 may be coupled to one or more of the boom assembly 170 and the frame 110, in a direction oriented towards the attachment 105, or the direction of the attachment 105. In one embodiment, the image sensor 195 may comprise of one or more cameras coupled to portions of the frame, or other immoveable parts of the work machine 100, and toggle between cameras as the boom assembly 170 moves to acquire a seamless image of the attachment 105. In another embodiment, the image sensor 195 may be coupled to the boom assembly 170, a moveable part of the work machine, to view the attachment.
The image sensor 195, generating a sensed input 270, may give a line-of-sight toward the attachment 105 or ground surface 135, and objects 85 around the work machine 100. The image sensor 195 may be utilized to detect objects 85 within a certain detection distance of the work machine 100. In one embodiment, the detection distance may be determined by the capabilities of the image sensor 195. In normal operation, the image sensor 195 may be configured to detect an object 85 closer than a distance threshold 285 from either the work machine 100 or the image sensor 195 itself. The distance threshold 285 may be pre-set or adjustable to avoid the anticipated/known ground surface irregularities from setting off the image sensor 195. Image sensor 195 may also be configured to require a detected object 85 be larger than a threshold size 290 before being considered an object 85, and this threshold size 290 may be pre-set or adjustable, based on the distance to the object from a reference point 295 or reference plane. In one exemplary embodiment, the reference point 295 may be a portion of the work machine 100, such as the frame 110, the boom assembly 170, the attachment coupler 185, or the attachment 105. Alternatively, the reference point 295 may be a point where the ground-engaging mechanism 155 engages the ground surface 135. In yet another alternative embodiment, the reference point 295 may be the image sensor 195 itself, or a receiving counterpart to the image sensor 195.
Image sensors 195 may be communicatively coupled to a processor 88 on the work machine 100 (alternatively, the processor 88 may be a part of the image sensor 195 itself or a worksite control center 280) that analyzes the sensed input 270 to determine whether an object 85 is present in the area and then communicates an object signal 260 indicative of the presence of an object 85 to a display 265. In one exemplary embodiment, the object signal 260 derived from the sensed input 270 from the image sensor 195 may be a value which indicates the absence of an object 85 (e.g. 0) or the proximity of the object 85 to the image sensor 195 (e.g. 1, 2, or 3 as the proximity increases). In alternative embodiments, the object signal 260 from the image sensor 195 may not itself communicate the presence or absence of an object 85 in an area but may instead communicate a value representative of the signal strength. In another embodiment, the object signal 260 may be derived from the dimensional attributes of an image where a distance and/or size of an object 85 may be calculated based on the known reference point 295 by the processor 88. The processor 88 may be communicatively coupled to the image sensor 195 to process the sensed input 270 into an object signal 260. In one embodiment, the processor 88 may be configured to monitor the object signal 260 in real-time to detect an object 85.
The object 85 may be in the path of travel 325 of the attachment 105. The processor 88 may determine a distance between the object 85 and a distance threshold 285, wherein the distance threshold 285 is a predefined distance for the controller 66 to recognize it is about to engage with an object 85 and therefore alter one of the speed of the work machine 100, and a position of the implement 105. The image sensor 195 may communicate other data to allow the controller 66 to interpret whether an object 85 is present in the area. Image sensor 195 may communicate further information such as the size of, distance to, or movement of the detected object(s), to enable the controller 66 to take different actions based on the size, distance, or movement of the detected object(s) 85. This information can be pictorial image, a simple camera image, or a combination of both.
As previously mentioned, the attachment 105 may be coupled to one or more of the boom assembly 170 and the frame 110. Within the application of the object detection system 200, the attachment 105 may be stationary or may be moving. For example, the forklift 205 shown is coupled to a front-end section 120 of the work machine 100. A mast 207 is a post coupled to a front surface of the frame 110, and its axis extends in an up and down direction. The fork is mounted to the mast 207 by being able to move in the up and down direction. Further, the fork is capable of swinging with respect to the mast 207 by a tilting mechanism in the direction of tilt 130. The fork includes a pair of tines 305 (shown in
In another exemplary application the attachment 105 is a hitch. The hitch 210, may be mounted on a rear portion of the work machine 100 to couple the work machine to another work machine, a trailer, or tool. In one exemplary embodiment, the hitch 210 may be raised by a piston movement of a hydraulic cylinder (not shown) when hydraulic oil is supplied into the hydraulic cylinder by a hydraulic pump (not shown). The hitch 210 may be a single point hitch. In another embodiment, the hitch 310 may be a three-point hitch including an upper link and a lower link. These are a few of several industry standard hitch configurations available with the use of work machines.
Depending on the application of the object detection system 200 and the work machine the object detection system is coupled to, the image sensor 195 may be one or more of forward facing and rear facing. However, alternative embodiments are not limited to either of the two directions. For example, a work machine, such as an excavator with an ability to rotate an attachment about a vertical axis 360 degrees, may have image sensors 195 in multiple direction if coupled to the base frame.
Processing of the object signal 260 include one or more of recognizing an object 85 in the image 282 (derived from the sensed input 270 and shown on a display 265) and defining the recognized object 337 as a target object 335. The processor 88 may define a bounded area 345 in the image 282 around the target object 335 and operate the work machine 100 wherein the object detection system 200 is configured to execute a function 350 of the work machine 100 when the target object 335 is defined. For example, the bounded area 345 may include a perimeter of the object 85, an area around the object or merely the intended contact area 360. The display device 265 may show an icon 332 representing the recognized object. In the display device 265, an icon 332 is an image that represents an object, an application, a capability, or some other concept or specific entity with meaning for the operator. This can include an image of the object itself, or an altered image representative of said object.
The controller 66 may be communicatively coupled to the processor 88 wherein the controller 66 sends a control signal 365 to one or more of a machine control system and the attachment control system to modify one or more of the movement of the attachment 105 and movement of the work machine 385 based on the object 85 reaching the distance threshold 285. In one instance, the attachment 105 may be powered by the work machine 100 and thereby be controlled by the machine control system. Alternatively, it may be self-powered through it’s own power source such as an battery and controlled through an attachment control system.
The processor 88 subsequently overrides execution of a function 350 of the work machine 100 when the object is defined as the target object. A function 350 of the work machine 100 may include one of alerting the operator 351, stopping the work machine 155, modifying a current travel speed of the work machine 155, and steering the work machine 155. Another function may include modifying the movement of the work machine 100 including one or more of several work machine parameters. The first may be modifying a speed of one or more of the left ground-engaging mechanism and the right ground-engaging mechanism of the work machine 100. For example, upon identifying a target object 335, the work machine 100 may begin slowing down for object engagement type applications. The relative motions of both the left ground-engaging mechanism and the right ground-engaging mechanism 155 can also translate into a degree of a change in direction of the work machine. Another may include pausing the work machine 392, thereby halting potential collision. Another may include modifying an acceleration 393 of the work machine 100. In the first embodiment of a skid steer, the work machine 100 may also modify the pitch 140 of the boom arms 190 to position an attachment for coupling to an intended contact area 360.
Modifying the movement of the attachment 105 comprises one or more of several other parameters 390 including pausing movement, and modifying the roll 130, yaw 140, and pitch 140, to name a few.
Now turning to
As used herein, “e.g.” is utilized to non-exhaustively list examples, and carries the same meaning as alternative illustrative phrases such as “including,” “including, but not limited to,” and “including without limitation.” As used herein, unless otherwise limited or modified, lists with elements that are separated by conjunctive terms (e.g., “and”) and that are also preceded by the phrase “one or more of,” “at least one of,” “at least,” or a like phrase, indicate configurations or arrangements that potentially include individual elements of the list, or any combination thereof. For example, “at least one of A, B, and C” and “one or more of A, B, and C” each indicate the possibility of only A, only B, only C, or any combination of two or more of A, B, and C (A and B; A and C; B and C; or A, B, and C). As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, “comprises,” “includes,” and like phrases are intended to specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof.
