Patent Application
20210140147
The present disclosure generally relates to a working machine, wherein the working machine is provided with an image projection arrangement specifically adapted for projecting an image at a surface of an external component comprised with the working machine. The present disclosure also relates to a corresponding method and computer program for such a working machine.
Working machines, in the form of e.g. excavators, dozers, loaders, and the like, may be used to perform various tasks in construction, mining and/or landscaping operations. Such working machines may generally be equipped with one or more work implements that can engage various objects to assist in performing the tasks. The work implements may be adjustable or manipulated in an intended manner by an operator, who e.g. may be positioned on-board the working machine. There exist various control systems and technologies to be incorporated into such working machines for assist its operator in controlling the working machine and/or work implements. These systems and technologies can interact with various a multitude of sensors for providing information and data to the operator regarding the working machine and/or work implements to further assist in control of the same.
Recent advances in computer science now allows the above mentioned control systems providing assistance to the operator to, in real-time, correlate data acquired by the sensors with a dynamic structural environment in which the working machine is operating. Based on the correlation, a so-called augmented reality (AR) may be displayed to the operator for assisting the operator in manipulating the working machine.
Specifically, AR may provide the operator with a view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, text, graphics, or video. In a typical prior-art AR application, a data processor reviews a camera-captured image for cues that trigger the display of additional information and images along with a display of the captured image.
The AR information may for example be presented to the operator using a display screen arranged within an operator compartment comprised with the working machine. However, such an implementation makes in necessary for the operator to split his attention between the display and e.g. the work implement, making the operation precarious and tiresome.
A more dynamic approach may as an alternative be taken, such as disclosed in US20140188333, where the operator is provided with a head mounted display (HMD) that is adapted for augmented reality capabilities. According to US20140188333, AR information may additionally be allowed to assist the operator to “see” the work implement even in cases where the work implement is obstructed from the direct view of the operator.
Even though the solution presented in US20140188333 provides great improvements to the safety operation of a working machine, the present inventors have identified further possibilities to assisting the operator, without having to resort to forcing the operator to wear e.g. a HMD, allowing further risk reductions when operating a working machine at e.g. a construction site or the like.
According to an aspect of the present disclosure, the above is at least partly alleviated by working machine comprising a working machine structure, working machine means pivotally connected to the working machine structure, and an operator compartment provided at the working machine structure and provided with control means adapted to allow an operator to control a position of the working machine means, wherein the working machine further comprises an image projection arrangement for projecting an image at a surface of the working machine means facing the operator compartment and the projected image comprises information for the operator to control the working machine.
In accordance to the present disclosure, it is thus possible to allow for any projection of information for assisting the operator to take place at an area externally of said working machine, i.e. without having to resort to presenting the information at a fixed or a mobile (e.g. HMD) display as proposed in accordance to prior-art. Rather, the working machine means comprised with the working machine is instead used as a “canvas” for presenting the assisting information.
An advantage following this implementation is that the operator of the working machine may be allowed to place his focus at the “right place”, which typically will be outside of the operator compartment rather than within the operator compartment. In addition, the projected information may typically be projected where and when “needed” and without the need to provide the operator with e.g. a dedicated HDM or similar. In addition, the solution proposed by the present disclosure solves prior-art problems of focal depth, where the operator has to re-focus when switching his gaze between the working machine means and the display device arranged within the operator compartment.
The image projection arrangement should be understood to mean an arrangement or a device that is capable of providing a visible illumination, including for example markings, lines, symbols, icons, text, images, etc., at the surface of the working machine means facing the operator compartment. The image projection arrangement may as such comprise different type of light sources, such as e.g. light emitting diodes (LED), or an array of LEDs, etc. Other alternatives are of course conceivable, such as e.g. lasers, image/video projectors, etc. Using a laser projector may in some embodiments be preferred due to the possible to provide a high focused monochrome light beam for projecting the image at the surface of the working machine means, as compared to what is some implementations may be achieved using a less focused light beam from e.g. an image projector. It should however be understood that the use of an image projector may be suitable in some implementations of the present disclosure.
Furthermore, the working machine may for example be at least one of an excavator, a wheel loader, a dozer, a grader and a backhoe loader, or any form of corresponding construction equipment comprising suitable working machine means arranged externally and at the same time visible for the operator and suitable for allowing the image to be projected in a manner as discussed above. Correspondingly, the working machine means may of course depend in the type of working machine. For example, the working machine means may in some embodiments comprises at least one of an implement and linkage. The implement may for example be a bucket or similar.
The working machine may, in some embodiments, comprise an internal combustion engine, or be at least one of a pure electrical vehicle (PEV) and a hybrid electric vehicle (HEV). Furthermore, in some embodiments of the present disclosure it may be desirable to equip the working machine with a geolocation arrangement, such as for example a GLONASS or GPS receiver comprised with the working machine, for determining a present location of the working machine such as at a work or construction site. Information relating to the position of the working machine may also comprise information relating to a relative orientation of the working machine.
In an embodiment of the present disclosure the image projection arrangement further comprises a control unit adapted to adjust a position of the image projected at the surface of the working machine means based on an expected position of the user when the user is positioned at the operator compartment. Such an estimation may for example be performed using an eye and/or head tracking mechanism connected to the control unit. With the estimated position available, it may be possible to allow e.g. perspective views of the image to be correctly projected when seen from the operator's present position. The estimation may be performed by determining an essentially exact location of the operator or the head/eye of the operator, or alternatively by e.g. estimating that the operator is located in a seat arranged within the operator compartment comprised with the working machine.
The control unit may in one embodiment be an electronic control unit (ECU) is typically provided as an onboard component of the working machine. Furthermore, the control unit is preferably adapted to receive information indicative of the current position of the working machine means (relative to e.g. the operator compartment or a ground surface), and to adapt the image projected at the working machine means based on the current position of the working machine means and/or the position of the operator.
The image projected at the surface of the working machine means may comprise information of a current or future task to be performed by the operator of the working machine, such as for example how to control the working machine means at e.g. the present location of the working machine. Such information may for example comprise a specific task using the implement for the excavator, such as digging a hole in ground at a specific location. Thus, the image formed and projected may include information for the operator of where the digging is to take place (“digging instructions”). The digging instruction, or similar, may also comprise a warning to the operator as to e.g. tubes and or pipes arranged underground. Thus, with such information at hand in may be possible to reduce possible digging accidents. The information may also be used informing the operator of e.g. at least one of a desired and a maximum filling level of the bucket.
It should however be understood that the information provided to the operator also, or instead, may include e.g. driving instructions to the operator for maneuvering the working machine. Alternatively, the information may be adapted to provide the operator with feedback on operation of the working machine, such as for example for operating the working machine in a more fuel-efficient manner, etc.
In addition to the above, the control unit may be further adapted to receive geographical data relating to a surrounding of the working machine. Such geographical data may possibly be acquired using one or a plurality of sensors arranged at or in the vicinity of the working machine. Such a sensor or sensors may for example include at least one of camera, a laser scanner, a radar arrangement, an IR based arrangement and a Lidar arrangement. The geographical data may provide information relating to e.g. how the ground surface is arranged, relate to possible structures or obstacles in the surrounding of the working machine, etc. The geographical data may, in one embodiment, be included when forming the image to be projected at the surface of the working machine means.
According to another aspect of the present disclosure there is provided a method for assisting an operator of a working machine, wherein the working machine comprises a working machine structure, working machine means pivotally connected to the working machine structure, an operator compartment provided at the working machine structure and provided with control means adapted to allow the operator to control a position of the working machine means, an image projection arrangement for projecting an image, and a control unit for controlling the image projection arrangement, wherein the method comprises the steps of determining, using the control unit, a position of the working machine means, and projecting the image at a surface of the working machine means facing the operator compartment based on the determined position of the working machine means. This aspect of the present disclosure provides similar advantages as discussed above in relation to the previous aspect of the present disclosure.
It should be understood that the concept according to the present disclosure may be implemented as a computer program, adapted to e.g. be executed using a control unit comprised with the above discussed working machine. In addition, such an implementation of the present disclosure provides similar advantages as discussed above in relation to the previous aspects of the present disclosure. The computer program may for example be stored onto a computer readable medium, where the computer readable medium may be any type of memory device, including one of a removable nonvolatile random access memory, a hard disk drive, a floppy disk, a CD-ROM, a DVD-ROM, a USB memory, an SD memory card, or a similar computer readable medium known in the art.
Further advantages and advantageous features of the present disclosure are disclosed in the following description and in the dependent claims.
With reference to the appended drawings, below follows a more detailed description of embodiments of the present disclosure cited as examples.
In the drawings:
The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the present disclosure are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness, and fully convey the scope of the disclosure to the skilled addressee. Like reference characters refer to like elements throughout.
With particular reference to
The load unit arrangement 102 is connected to the upper structure 108 of the excavator 100, which makes it rotatably movable relative to the lower structure 110 as well as relative to the ground surface 118 thereof. The load unit arrangement 102 can also be lifted and lowered relative to the ground surface 118 such that the distance between the implement 106 and the ground surface 118 increases/reduces. In addition, the implement 106 can be tilted relative to the boom arm 104. As further illustrated in
As described, the load unit arrangement 102 is movable relative to the lower structure 110, as well as relative to the ground surface 118.
The working machine 100 further comprises image projection arrangement 200. A detailed description of the image projection arrangement 200 and its implementation and operation in relation to the working machine is given in relation to the description below.
Turning now to
The image projection arrangement 200 can as discussed above comprise e.g. a laser projector 204 for presenting information at the surface 122. The image projection arrangement 200 further comprises a control unit 210, such as an electronic control unit (ECU), adapted to form the image to be projected by the laser projector 204. The control unit 210 is further adapted to control the laser projector 204. The control unit 210 may for example be manifested as a general-purpose processor, an application specific processor, a circuit containing processing components, a group of distributed processing components, a group of distributed computers configured for processing, a field programmable gate array (FPGA), etc.
The processor may be or include any number of hardware components for conducting data or signal processing or for executing computer code stored in memory. The memory may be one or more devices for storing data and/or computer code for completing or facilitating the various methods described in the present description. The memory may include volatile memory or non-volatile memory. The memory may include database components, object code components, script components, or any other type of information structure for supporting the various activities of the present description. According to an exemplary embodiment, any distributed or local memory device may be utilized with the systems and methods of this description. According to an exemplary embodiment the memory is communicably connected to the processor (e.g., via a circuit or any other wired, wireless, or network connection) and includes computer code for executing one or more processes described herein.
The control unit 210 is further adapted to receive information indicative of a present location of the working machine 100, such as from a geolocation arrangement 212 comprised with the working machine 100. The geolocation arrangement 212 may for example comprise a GPS receiver 212 or a local positioning arrangement at a construction site, such as for example a Wi-Fi positioning system. It may, as understood by the skilled addressee, be necessary to use an alternative to satellite navigation in case the working machine is operating underground, such as for example in a mine or similar.
In addition, the control unit 210 may be adapted to receive information relating to an orientation of the working machine 100, relative to e.g. the work site, thus allowing the working machine 100 both positioned and oriented in a desired manner. Such orientation information may for example be provided by the GPS receiver 212 or using a separate compass functionality provided with the working machine 100. In the exemplary implementation shown in
The image projection arrangement 200 may further comprise a database 214 arranged onboard the working machine 100 or remotely from the working machine 100. The database 214 is arranged in communication with the control unit 210. In case the database 214 is arranged remotely from the working machine 100 the image projection arrangement 200 may comprise a transceiver (not shown) used for establishing a network connection with the database 214. The database 214 may for example comprise digging instructions, driving instructions or similar relating to a specific location (e.g. working site) where the working machine 100 is to be operated.
In addition to the above, the control unit 210 is preferably adapted to receive sensor data relating a surrounding of the working machine 100, such as from a camera arrangement 216. The camera arrangement 216 may, in some embodiments, be a three-dimensional (3D) camera adapted to be used for forming a 3D representation of the surrounding of the working machine 100. Accordingly, the control unit 210 is adapted to implement a feedback functionality where e.g. sensor data may be correlated with e.g. the digging instructions, driving instructions or similar stored within the database 214. Thus, it may be possible to determine a matching between a current state at e.g. the work/construction site (i.e. collected using the camera arrangement 216) and the “desired” end result provided by the construction data. The correlation will further allow the projected image to be updated in a corresponding manner, i.e. to be continuously dependent on the current status of the work/construction site and the operation of the working machine means performed by the operator. The control unit 210 may also be adapted to be connected to navigation equipment (not shown) comprised with the working machine 100 for providing navigation instructions to the control unit 210 as will be further discussed below.
The control unit 210 may additionally be adapted to (continuously) receive information relating to a current positioning of the working machine means. Such information may for example be available at a CAN bus comprised with the working machine 100 or alternatively received from further sensors (not shown) arranged at the working machine means.
With further reference to
In
The instruction to be provided to the operator in accordance to
As a result of the projected image 302, showing an arrow pointing upward, the operator is to adjust the position/location of the implement 106 such that the implement 106 is raised, and the control unit will 210 will as a consequence determine, S3, an adjusted position of the implement 106. In case the implement 106 has been “raised more” than what was desired (e.g. raised “too much”), an adjusted, S4, image 304 may be formed and subsequently projected, S5, at the surface 122. In
As an alternative and as shown in
In
Furthermore and as indicated above, the image projected at the surface 122 may not necessarily be limited to show information relating to e.g. a specific handling of the implement 106. Rather, other types of information may be provided. As exemplified in
As indicated above, the image projected at the surface 122 may of course hold other information, such as a current weight of the implement 106 (when loaded), an instruction to operate the working machine 100 in a more energy efficient manner, etc. Accordingly, any type of suitable information that may be useful for operating the working machine 100 in an efficient manner may be provided to the operator at the surface 122.
The present disclosure contemplates methods, devices and program products on any machine-readable media for accomplishing various operations. The embodiments of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system. Embodiments within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor.
By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a machine, the machine properly views the connection as a machine-readable medium. Thus, any such connection is properly termed a machine-readable medium. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data that cause a general-purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.
Although the figures may show a specific order of method steps, the order of the steps may differ from what is depicted. In addition, two or more steps may be performed concurrently or with partial concurrence. Such variation will depend on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations could be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps and decision steps. Additionally, even though the disclosure has been described with reference to specific exemplifying embodiments thereof, many different alterations, modifications and the like will become apparent for those skilled in the art.
Variations to the disclosed embodiments can be understood and effected by the skilled addressee in practicing the claimed disclosure, from a study of the drawings, the disclosure, and the appended claims. Furthermore, in the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2017/064374 | 6/13/2017 | WO | 00 |
