Patent Application
20170185906
The present disclosure relates to systems and methods for determining the operational state of a machine in a worksite. The disclosure is particularity suitable for implementation in mining worksites and will be described in relation to that exemplary but non-limiting application.
Worksites are operated with one or more goals in mind. In order to achieve those goals, worksites use machines to perform various operations, with different operations typically involving a number of different operational states.
For example, a goal of mining worksites is to yield ore. One typical mining operation, therefore, is the transport of excavated material. For example, in open-cut mines material is excavated from worksite locations (e.g. faces) and loaded into haulage machines for transport to a designated dumping or processing area. In the course of this broad operation, machines pass through various operational states. For example, a haulage machine (in the operation of transporting material from where it is excavated to where it is stockpiled or processed) will typically pass through the operational states of: travelling empty to a loading site, queuing at the loading site, spotting, being loaded, travelling full to a dumping site, queuing at the dumping site, and dumping.
Accurately determining the particular operational state a machine is in can be useful. Such operational state information can be used to monitor/assess machine productivity, monitor/assess machine operator performance/productivity, estimate worksite productivity, detect or anticipate abnormal machine or worksite conditions.
U.S. Pat. No. 8,364,440, titled “System for evaluating the productivity of a working machine and its driver” describes identifying different work cycles of a working machine based on the use of the controls by the driver of the working machine (i.e. by the driver's use of the working machine's control stick).
In large working environments, however, difficulties with worksite communications can exist. For example, worksite machine sensors can be broken or faulty and/or clocks on different worksite machines may not be synchronised. Further, communications across the worksite can be variable, particularly where machines operate in worksite locations with poor coverage, which can lead to communications going missing, arriving out of time, being duplicated, and/or being corrupted. Such issues can impede the ability to accurately estimate machine operational states.
Disclosed herein is a computer-implemented method for estimating an actual operational state of a machine of interest at a point in time, the machine of interest being involved in a worksite operation comprising a plurality of different operational states, wherein the method comprises operating a computer processing unit to: receive a plurality of messages, each message comprising data relating to the machine of interest as it performs the worksite operation, the plurality of messages comprising data from at least two separate sources; process the data from the plurality of messages using a statistical process to generate an estimated actual operational state of the machine of interest at the point in time.
The at least two separate data sources may comprise a first data source comprising data derived from a first sensor carried by the machine of interest and a second data source comprising data derived from a second sensor carried by the machine of interest.
The at least two separate data sources comprise a first data source comprising data derived from the machine of interest and a second data source comprising data derived another worksite machine.
The at least two separate data sources comprise a data source of contextual information.
The plurality of messages may comprise at least one perceived state message comprising data relating to a perceived operational state of the machine of interest at a point in time.
The source of the at least one perceived state message may be the machine of interest.
The computer-implemented method may further comprise operating the computer processing unit to: process the estimated actual operational state against alert criteria; and responsive to the alert criteria being satisfied, raise an alert comprising at the estimated actual operational state of the machine of interest.
The computer-implemented method may further comprise operating the computer processing unit to: process the plurality of operation messages using the statistically based process to generate uncertainty information associated with the estimated actual operational state of the machine of interest at the point in time, the uncertainty information defining an uncertainty with respect to the estimated actual operational state of the machine of interest at the point in time, and wherein the uncertainty information associated with the estimated actual operational state is used in the processing of the estimated actual operational state against the alert criteria.
The computer processing unit may be operated to: receive the plurality of messages in real time; and process the plurality of messages in real time such that the estimated actual operational state of the machine of interest is generated in real time.
The computer implemented method may further comprise operating the computer processing unit to: generate a computational model of the worksite operation, the computational model comprising the plurality of different operational states and plurality of observations relevant to the plurality of operational states, wherein each of the plurality of messages comprises data relating to an observation, and wherein processing the data from the plurality of messages using the statistical process comprises processing the data using the computational model to estimate the actual operational state of the machine of interest.
The computer-implemented method may further comprise operating the computer processing unit to train the computational model using a training dataset.
The computational model may comprise a Hidden Markov Model and the operational states may be hidden states of the Hidden Markov Model.
Processing the data from the plurality of messages using the statistical process may comprise using a Viterbi Algorithm to process the data using the computational model to estimate the actual operational state of the machine of interest.
The worksite process may comprise operating haulage machines to haul material from a loading location to a dumping location, and the machine of interest may a particular haulage machine.
The plurality of different operational states may comprise: the machine of interest queuing at a loading location; the machine of interest spotting; the machine of interest loading; the machine of interest travelling full; the machine of interest queuing at a dumping location; the machine of interest dumping; the machine of interest travelling empty.
Also disclosed herein is a computer processing system configured to estimate an actual operational state of a machine of interest at a point in time, the machine of interest being involved in a worksite operation comprising a plurality of different operational states, the computer processing system configured to: receive a plurality of messages, each message comprising data relating to the machine of interest as it performs the worksite operation, the plurality of messages comprising data from at least two separate sources; operate a processing unit to process the data from the plurality of messages using a statistical process to generate an estimated actual operational state of the machine of interest at the point in time.
Also disclosed herein is a non-transitory computer-readable medium comprising instructions which, when implemented by a computer processing system, cause the computer processing system to estimate an actual operational state of a machine of interest at a point in time, the machine of interest being involved in a worksite operation comprising a plurality of different operational states, the instructions causing the computer processing system to: receive a plurality of messages, each message comprising data relating to the machine of interest as it performs the worksite operation, the plurality of messages comprising data from at least two separate sources; process the data from the plurality of messages using a statistical process to generate an estimated actual operational state of the machine of interest at the point in time.
As used herein, the term “comprises” (and grammatical variants thereof) is used inclusively and does not exclude the existence of additional features, elements or steps.
Various aspects and features will be described with reference to the following figures which are provided for the purposes of illustration and by way of non-limiting example only.
A number of machines 102 work on or at worksite 100. In this instance the machines 102 depicted comprise a loading machine 102a, haulage machines 102b, and levelling machines 102c (e.g. dozing or grading machines). Machines 102 operating at the worksite 100 may be directly controlled by human operators, remotely controlled by human operators, be autonomous machines capable of autonomously working and traversing the worksite 100, or be semi-autonomous machines configured to perform some functions autonomously and other functions under the control of an operator.
Worksite 100 also comprises a number of locations designated for particular purposes. In this example the locations comprise: a loading location 104 designated as a location at which a loading machine 102a or other resource operates to fill haulage machines 102b with material; a dump location 106 designated as a location at which haulage machines 102b drop collected material; a refuelling location 108 designated as a location which mobile machines can attend to be refuelled; and a maintenance location 110 designated as a location which mobile machines can attend to be maintained. Worksite 100 also comprises designated roads or paths 112 (each of which may have multiple lanes) linking various worksite locations and along which mobile worksite machines can travel.
Worksite 100 may also comprise various dedicated monitoring systems 116 which perform worksite monitoring. For example, worksite 100 is depicted with a weigh bridge 116a (which can weigh machines 102 as they traverse the bridge 116a) and a dedicated worksite camera 116b which captures visual data (still or video) of an area of the worksite 100.
Worksite 100 also comprises a radio communications tower 118 operated to relay messages between various worksite machines 102, monitoring systems 116, and a control centre 120. Control centre is shown in this in this instance as being remote from the worksite 100, however could equally be situated on the worksite 100. A given worksite 100 may well have a number of radio communications towers 118 to increase communications coverage and reliability over the worksite 100.
It will be appreciated that the types of machines 102, monitoring equipment 116, designated locations 104, 106, 114 described above are by way of illustration only. A given worksite may well have additional (or fewer) machines 102 and/or monitoring equipment 116, different types of machines 102 and/or monitoring equipment 116, and/or different designated locations.
Turning to
Generally speaking, the control centre 120 performs a central monitoring and control function for the worksite 100. This function is achieved by receiving information in respect of the worksite, processing it, and managing worksite operations accordingly. To enable this control centre 120 is provided with a computer processing system 202.
Computer processing system 202 (in the present example) comprises at least one processing unit 204 which may be a single computational processing device (e.g. a microprocessor or other computational device) or a plurality of computational processing devices. Through a communications bus 206 the processing unit 204 is in data communication with a system memory 208 (e.g. a read only memory storing a BIOS for basic system operations), volatile memory 210 (e.g. random access memory such as one or more DRAM modules), and non-transient memory 212 (e.g. one or more hard disk drives, solid state drives, flash memory devices and suchlike). Instructions and data for controlling operation of the processing unit 204 are stored on the system, volatile, and/or non-transitory memory 208, 210, and 212.
The computer processing system 202 also comprises one or more input/output interfaces 214 which allow the system 202 to interface with a plurality of input/output devices 216. As will be appreciated, a wide variety of input/output devices may be used, for example keyboards, pointing devices, touch-screens, touch-screen displays, displays, microphones, speakers, hard drives, solid state drives, flash memory devices and the like. Computer processing system 202 also comprises one or more communications interfaces 218, such as a Network Interface Card, allowing for wired or wireless connection to a communications network 220 (e.g. the Internet, a local area network, or a wide area network) and communication with other computer processing systems connected to the network 220.
Communication with the communications network 220 (and other devices connected thereto) will typically be by the protocols set out in the layers of the OSI model of computer networking. For example, applications/software programs being executed by the computer processing system 202 may communicate using one or more transport protocols, e.g. the Transmission Control Protocol (TCP, defined in RFC 793) or the User Datagram Protocol (UDP, defined in RFC 768). Alternative communications protocols may, of course, be used.
The computer processing system 202 stores in memory and runs one or more applications allowing operators to operate the device system 202. Such applications will typically comprise at least an operating system such as Microsoft Windows®, Apple OSX, Unix, or Linux.
Each worksite machine 102 is also equipped with at least one, and typically multiple, computer processing system(s) 230. Often, a given machine 102 will be equipped with multiple processing systems for performing multiple tasks. For example, a machine 102 may be equipped with a machine vital information management system (e.g. VIMS system as provided by Caterpillar™), a dedicated payload monitoring system, a dedicated tyre pressure monitoring system, and an operator display/control system. Each of these systems may be interconnected so as to be able to communicate with each other and/or be configured to communicate independently with an external system (e.g. with control centre 120 via network 220). For ease of reference the various computer processing systems of machines 102 will simply be referred to as machine computer processing system 230, but it will be appreciated from the foregoing that machine processing system 230 may, in fact, be multiple independent or interconnected processing systems. The computer processing system(s) 230 of each machine 102 may be similar in some respects to computer processing system 202 described above, though will typically be a dedicated system tailored specifically for the machine and/or operation in question. Worksite machines 102 are also typically equipped with a plurality of sensors 232 which may be integral with the various computer processing systems of the machine 102, or be separate components that interface with computer processing system(s) 230 of the machine, for example via input/output interfaces 214. The sensors 232 for a given machine typically comprise a number of different types of sensors (and, at times, multiple sensors of the same type) which are operated to sense and detect information regarding the operation of the machine and, in some cases, the environment proximate to the machine.
By way of non-limiting example, the sensors 232 for a machine 102 may comprise: a Global Positioning System (GPS) device, an Inertial Measurement Unit (IMU), radar, lidar, range sensors, video cameras, a fuel level sensor, a fuel pump sensor, payload load sensors, tyre pressure sensors, temperature sensors, a speed sensor, a gear sensor, tool position and/or orientation sensors (e.g. detecting for a haulage machine 102b whether the tray/bed is up or down, or detecting in a loading machine 102a the orientation/position of the dipper). In addition, machines 102 are typically equipped with more sophisticated sensor/monitoring systems (each of which may, for example, be an independent or interconnected computer processing system as discussed above) for monitoring major machine subsystems such as: machine engine; machine transmission; machine suspension; machine tyres; machine steering; machine exhaust; machine electrical and suchlike.
Typically, the sensors 232 of a given machine are controlled by and communicate sensed data to the computer processing system 230 of the machine. The computer processing system 230 processes the raw sensed data to generate predefined messages for transmission to the control centre 120 either in real-time or batch mode. For example, the computer processing system 230 of a machine 102 may periodically process available positional data (obtained, for example, from the GPS, IRU, and any other positional sensors) and generate a position report message reporting on the machine's position in the worksite 100. These messages are then transmitted to the computer processing system 202 of the control centre 120 (and, optionally, stored on a local memory of the computer processing system 230 of the machine 102). Messages between machines 102 and the control centre 120 are discussed in further detail below.
The computer processing system 230 of each machine 102 also enables the machine 102 to receive messages from the control centre 120 (e.g. via a communications interface 218). These messages can comprise, for example, task assignment messages with instructions which define new tasks (or modifying existing tasks) for the machine 102. Such messages can be displayed or otherwise imparted to an operator of the machine 102 (e.g. via output devices such as displays or and or speakers), or where the new/amended task can be carried out without operator intervention be processed by the computer processing system 230 for automatic implementation by the machine 102.
Each dedicated monitoring system 116 of the worksite 100 will also be equipped with (or be) a computer processing system 240. As with the worksite machines 102, the computer processing system 240 of a given monitoring system 116 will be similar in some respects to computer processing system 202 described above, and has been depicted that way in
As with the worksite machines 102 described above, the sensors 242 of a given monitoring system are controlled by and communicate sensed data to the computer processing system 240, which can then communicate the sensed data to the computer processing system 202 of the control centre 120.
It will be appreciated that the foregoing example of the control centre 118 computer processing system 202 (referenced also in the context of machines 102 and monitoring systems 116) is provided by way of non-limiting example only. A wide variety of digital processing systems with different components and/or architectures could be used for either the control centre 120, individual machines 102, and/or individual monitoring systems 116.
In addition, although only one vehicle 102 and one monitoring system 116 are depicted in
As noted above, during operation of the worksite 100 information is collected from a wide variety of sensors. Messages (which will be referred to as incident messages) are communicated to the computer processing system 202 of the control centre 120. To illustrate the number of messages (and amount of data) being processed, and recognising that all worksites are, of course, different, a typical open cut mine worksite may cover an area of up to (or over) 100 square kilometres and have 1000 or more machines operating simultaneously. By way of estimate, half of the machines may be “heavy” machines (such as loaders, dozers, graders, haulage trucks, draglines and the like), and half or the machines may be “light” machines (e.g. transportation vehicles and suchlike). Each machine may send approximately 1 to 2 messages per second, which can provide for the generation of up to (or over) 2000 incident messages per second.
At the computer processing system 202 of the control centre 120 incident messages (or, at least, the information communicated by the incident messages) are stored in one or more databases maintained, for example, in non-volatile memory 212. The information from the incident messages is made available for use (e.g. access) by programmatic applications and human operators in order to perform various worksite management functions.
A number of issues with the data gathered from the worksite can exist. Such issues can arise, for example, from: broken sensors leading to expected incident messages not being received; faulty or miscalibrated sensors leading to incident messages comprising inaccurate information; incident messages becoming corrupted; incident messages being delayed; incident messages being received out of order; incident messages being duplicated.
Further issues can arise where incident messages report seemingly contradictory information. For example, in the worksite 100 multiple machines 102 may report on a shared activity from different perspectives. As one simple example, a loading machine 102a may report that it is currently loading a particular haulage machine 102b. A different haulage machine, however, may report that it is currently loading. This generates a mismatch between the observations of the loading machine and haulage machines, making it difficult to determine which particular haulage machine is actually being loaded.
As another example, a loading machine 102a may send an incident message reporting the commencement of a loading operation at a certain time t1, a haulage machine 102b which is (in reality) receiving the material from the loading machine 102a may send an incident message reporting the commencement of the loading operation at an alternative time t2, and a camera monitoring system 116b may send visual data that depicts the loading operation in question commencing at a further alternative time t3.
This, in turn, illustrates a more general issue that can arise with worksite information. Time is often a useful dimension for sorting and analysing data (i.e. the chronological sequence of operations that have occurred/are occurring at the worksite). Given the number of machines 102 and monitoring systems 116 operating at the worksite it is generally the case that the internal clocks of various machines 102 and/or monitoring systems 116 are not synchronised. This can complicate the task of trying to generate an accurate timeline of worksite activity and events.
On order to run worksite 100, machines 102 are assigned with a variety of high-level operations. Such operations may comprise, for example, operations directed at altering the geography of the worksite 100, operations directed at recovering material from the worksite, operations directed at maintaining the worksite and so forth.
For example one operation may be a material transport operation, involving the transport of material from one location (e.g. the loading location 104) to another (e.g. the dump location 106). From the perspective of a loading machine 102a this operation generally comprises the operational states of: awaiting the arrival/positioning of a haulage machine 102b; collecting material at the loading location 104; loading the collected it into haulage machines 102b. From the perspective of hauling machines 102b, this operation generally involves attending/queuing at the loading location 104, manoeuvring into position to receive a load of material from a loading machine 102a, traversing roads 112 with a full load to the dump location 106, dumping the load of material at the dump location 106, and traversing roads 112 empty back to the loading location 104.
Information as to the particular operational state of a specific machine 102 at a given time is recorded by system 202 (e.g. in a suitable data structure on memory such as non-transient memory 212). This information is useful as it can be analysed to generate information relevant to the performance of the worksite 100, the performance of individual worksite operators/machines, operational faults of worksite machines, and/or worksite conditions.
To this end, certain worksite machines 102 (e.g. haulage machines 102b) are configured to estimate their own current operational states (i.e. a self-perceived state) and communicate messages comprising those self-perceived states to the control centre 120. Due to the problems described above, however, such estimates may not always be accurate—i.e. the self-perceived operational state of a given machine 102 may not match is actual operational state. Further, and whether a machine's operational state estimate is accurate or not, operational sate messages may go missing, be corrupted, have inaccurate time-stamps, and/or be delivered out of order.
To assist in more accurately determining the operational states of machines 102, system 202 of the control centre 120 is configured to receive a plurality of incident messages from various machines 102 and process those message using a statistically based method in order to provide a more accurate estimate of the operational state of the particular machine 102.
In one embodiment the configuration of system 202 is achieved by software which comprises instructions which can be executed by a processing unit 204 to implement the relevant features of the method. The software/instructions will typically be stored on a non-transient computer-readable medium, such as a non-transient memory 212 or an external data storage device which can interface with the computer processing system 202 via, for example, an input/output interface 214. The software/instructions may be provided to computer processing system 202 by means of a data signal in a wired or wireless transmission channel over communications interface 218. In alternative embodiments the system may be implemented as hardware circuitry configured to implement the processes described.
In order to describe the principles of the disclosure, the example of estimating the operational state of a haulage machine 102b during a material transport operation will be described. It will be appreciated, however, that the principles could equally be applied in order to estimate the operational states of other machines involved in the material transport operation, and/or the operational states of machines 102 involved in alternative operations.
The statistical method implemented by system 202 makes use of a computational model which, in this example, is a Hidden Markov Model (HMM). Alternative Multi-State Modelling techniques could be used, for example other Bayesian Inference techniques based on Markov Chain Monet Carlo Simulations (such as Particle Filters). The process of generating and training the computational model will be described with reference to
At 302, operational state information defining the operational states (402a-402g in
At 304, information on possible state transitions (404a-404f in
The operation in question may be cyclical. In the present example machines will often transition from travelling empty (state 402g) back to queuing at loader (state 402a). There may, however, also be non-productive operations or states that occur that (at least in this case) are not modelled. Non-productive operations may comprise, for example refuelling operations, maintenance operations, and shift change operations.
At 306, initial transition probabilities are calculated. The transition probabilities indicate the likelihood that the system will move from one state to another. A variety of options exist for this initial calculation, within the general constraint that the sum of the transition probabilities emanating from a given state is 1.0. For example, system 202 may be configured to: randomly calculate initial state transition probabilities; calculate initial state transition probabilities based on received information regarding the operation in question (e.g. sample data regarding its historical performance and/or anticipated future performance); calculate initial state transition probabilities such that each possible transition from a given state has the same transition probability (i.e. for a given state having x possible transitions, the state transition probability for each transition from that state is calculated to be 1/x).
Information on the states 402 and state transition probabilities is stored in a state transition matrix, stored in turn on a memory device accessible by processing unit 204 (e.g. non-transient memory 212). Table 1 provides an example state transition matrix recording initial state transition probabilities for the current haulage machine material transport operation example. For the purposes of the present example, the methodology of calculating each initial state transition probability from a given state to be equal has been adopted. As each state has only one possible transition each initial state transition probability is (in this particular instance) 1.0. As the model is trained (discussed below) the initial transition probabilities are refined.
At 308, types of information that will (or should) be received by system 202 and that are relevant to the operation in question are received. Relevant information to the operation is determined by an operator according to the various incident messages that are expected to be received from the worksite 100 and which are of potential relevance to determining any of the operational states 402. These are input into/uploaded to system 202 by the operator, and are treated as observations of the HMM (see 406a-406q in
As can be seen, the observations 406 in the present example comprise observations/information from a plurality of separate sources. Separate sources in this case can relate to observations/information derived from different worksite machines. For example, the queuing empty observation/information is sourced from the actual haulage machine 102b whose state is being estimated and the dipper report observation/information is sourced from a loading machine 102a (the haulage machine 102b and loading machine 102a being separate sources of information). Separate sources can also relate to observations/information derived from different sensors or different sensor groups on a given machine 102. For example, the stopping observation/information is sourced from a first sensor/group of sensors carried by the haulage machine 102b whose state is being estimated and the reversing observation/information is sourced from a second, different sensor/group of sensors carried by the haulage machine 102b whose state is being estimate (the first sensor/group of sensors and the second sensor/group of sensors being separate sources). One group of sensors should be considered different to another group of sensors provided there is at least one sensor present in one group and not the other. By way of further example, contextual information available to system 202 regarding the worksite and/or worksite operations may provide a separate source of information. Contextual information may comprise information such as assignment information for a machine in question (i.e. the task currently assigned to a machine), shift-change information (i.e. scheduled shift-change times and locations), map information (i.e. information as to the location of various worksite paths and locations).
The observations 406 also comprise observations made by the actual machine whose state is being estimated as to its own perceived state—e.g. the queuing empty observation 406b (being an observation made by the machine in question that it is in the queuing empty state), the reversing observation 406c, the loading observation 406d, the travelling full observation 406i, the queuing full observation 406j, the dumping observation 406k. These observations are obtained/derived from various sensor measurements. As described above, however, a machine's perception of its own state may be incorrect (e.g. due to faulty or miscalibrated sensors), and/or messages communicated to the control centre and including the self-perceived state information may be compromised which impacts on the reliability of these self-perceived state messages. By taking into account additional observations (both from the machine itself and other machines) system 202 can make a more reliable estimate as to the actual operational state of a machine.
At 310 emissions in respect of the model are generated. In the present embodiment the model is prepared on the assumption that any observation may be made from any state. In
At 312 initial emission probabilities are calculated (i.e. the probability of a given observation 406 occurring whilst in a given state 402). A variety of options exist for this initial calculation within the general constraint that the sum of the emission probabilities emanating from a given state is 1.0. For example, system 202 could be configured to: randomly calculate initial emission probabilities; calculate initial emission probabilities based on received information regarding the operation in question (e.g. information re the historical performance and/or anticipated future performance of the operation); calculate the initial emission probabilities such that each possible emission from a given state has the same emission probability (i.e. for a given state having y possible emissions, the emission probability for each emission from that state is calculated to be 1/y). Typically a rough estimate of emission probabilities (based on experience and judgement) will be entered by a user.
Information on the observations 406, emissions 408, and emission probabilities is stored in an emission probability matrix, stored in turn on a memory device accessible by processing unit 204 (e.g. non-transient memory 212). Table 2 provides an example of a partial emission probability matrix recording initial emission probabilities for the current haulage machine material transport operation example. For the purposes of illustration, the methodology of calculating initial emission probabilities from a given state to be equal has been adopted. Taking the queuing at loader state 402a, as there are 17 possible emissions each emission probability is initially calculated as 1/17.:
At 314 initial starting state probabilities for the HMM are calculated. Once again a variety of options exist for this initial calculation within the general constraint that the sum of the initial starting state probabilities is 1.0. For example, system 202 may be configured to: randomly calculate initial starting state probabilities; calculate initial starting state probabilities based on received information regarding the operation in question (e.g. its historical performance and/or anticipated future performance); calculate initial starting state probabilities such that each possible state has the same initial starting state probability (i.e. for an operation having z states, the initial starting state probability for each state is calculated to be 1/z).
Information on initial starting state probabilities is stored in a starting state matrix, stored in turn on a memory device accessible by processing unit 204 (e.g. non-transient memory 212). Table 3 provides an example starting state matrix recording initial starting state probabilities for the current haulage machine material transport operation example. For the purposes of the present example, the methodology of calculating initial starting state probabilities to be equal has been adopted. As there are 7 possible states, each initial starting state probability is calculated as 1/7.
At 316 system 202 trains the computational model. In order to train the HMM system 202 is, in the present embodiment, processes the computational model developed (as described above) and a training dataset in accordance with a Baum-Welch algorithm. The training dataset comprises observations from historical implementations of the operation (typically by many machines). Operating system 202 processes the training dataset according to the Baum-Welch algorithm to generate: updated state transition probabilities that reflect the training dataset (stored, for example, in an updated state transition probability matrix on a memory such as 212); updated emission probabilities that reflect the training dataset (stored, for example, in an updated emission probability matrix on a memory such as 212); updated starting state probabilities that reflect the training dataset (stored, for example, in an updated starting state matrix on a memory such as 212).
Generally speaking, the more information comprised in the training dataset the more accurate the HMM will be. Further, the computational model may be periodically retrained if necessary or desired as more observational data is obtained.
At 318, system 202 tests the trained HMM by processing a dataset using the trained HMM to estimate machine operational states based on the observations included in the dataset. In the present embodiment system 202 is configured to process the training dataset and trained HMM using a Viterbi algorithm.
By processing the training dataset and trained HMM using a Viterbi algorithm, regions where the model performs poorly are identified—for example by identifying numeric underflows and overflows in the computation. Where such underflows/overflows are identified, system 202 is configured to interpret this as indicating that the process has become non-stationary. In this case system 202 splits the problem into separate regions and processes each region separately (e.g. by separately processing each region using a Viterbi algorithm).
From the testing of the trained HMM at 318, system 202 calculates both operational state estimations based on the observations and, for each estimated operational state, an associated operational state uncertainty indicating the probability that the estimated state is accurate.
While
At 502, system 202 receives incident messages from the worksite 100. At 504, system 202 parses the incident message to extract observational data and machine identification information (identifying the particular machine that the observational data relates or is thought to relate to). Each item of observational data matches an observation 406 included in the model 400 generated in respect of the operation.
At 506, system 202 processes the observational data relevant to the machine in question using the trained HMM. In the present embodiment system 202 is configured to process the observational data and HMM using a Viterbi algorithm which generates an estimate as to the current operational state of the machine in question, together with an associated operational state uncertainty indicating the probability that the estimated state is correct.
At 508 system 202 processes the estimated operational state and associated uncertainty against alert criteria to determine whether an alert needs to be raised.
By way of example, the alert criteria may be that a predefined number of state estimates are made in a row with each state estimate having an uncertainty the exceeds a predefined threshold. E.g., if four state estimates are made each of which having an uncertainty of greater than 20% an alert is raised.
In the context of reporting on worksite operations, an alert may simply be raised where the uncertainty associated with an identified state exceeds a threshold (e.g. any identified state with an uncertainty of greater than 20% is flagged for review). Flagging uncertain states in this manner reduces the workload of human operators reviewing such reports, insofar as instead of checking all reported states to see if they are correct/appear sensible, only those states which are flagged need to be reviewed.
If the alert criteria are not satisfied, no alert is raised and process returns to 502 to receive/process further incident messages.
If the alert criteria are satisfied, system 202 raises an alert at 510. In the present embodiment system 202 is configured to raise an alert by displaying an alert message on a display for a control centre operator to view and action. Alerts may, of course, be raised to additional or alternative human operators and/or in additional or alternative ways. For example, depending on the operation in question and the alert to be raised, system 202 may alert a control centre operator, a worksite overseer, and/or one or more a machine operators. Human operators may be alerted by (again by way of non-limiting example): displaying an alert on a display; generating and sending an email to one or more email addresses; generating and sending a SMS to one or more phone numbers; generating and sending an instant message to one or more instant message addresses.
On receiving an alert, an operator takes the appropriate action. Typically this will be to review the currently recorded operational state of the machine in question and, if required, manually correct it. The operator may determine the actual state of the machine in question in a number of ways. For example, the operator may: observe real time footage of the machine (received, for example, by an appropriately positioned on-site video camera); contact the operator of the machine; contact another on-site operator with knowledge of the operational state of the machine in question.
In some embodiments, the trigger of an alert may result in additional analysis of worksite information being performed in order to determine the likely state of the machine in question. For example, if an alert is raised due to a low confidence that a haulage machine is loading at a nominated loading machine, the system 202 may process additional information (e.g. the locations of the loading and haulage machines, the proximity of the loading and haulage machines to one another, assigned destination of the haulage machine, any other potentially relevant worksite information) to try and determine the likely actual state of the haulage machine.
At 512, updated operational state information is received and recorded by system 202.
Process 500 can be performed in real time, in the sense that incident messages are processed as they are received from the worksite 100 and if an alert is to be raised it is raised as soon as that has been determined. Automatically flagging operational state alerts and allowing misidentified operational states to be corrected in real time provides for far greater efficiencies than having operators manually inspect the relevant data in batches (e.g. on a per-shift basis, which can involve hundreds of operational cycles) to identify and correct any errors. Identifying and correcting operational state errors in real time can also be beneficial as a misidentified operational state for a given machine can adversely impact the identification of future operational states for that machine, as well as the identification of operational states of other machines.
As noted above, machines may at various times perform non-productive operations (e.g. refuelling operations, maintenance operations, shift change operations) which need not necessarily be modelled. Non-productive operations can be accounted for either manually or automatically.
For example, when a machine commences a non-productive operation an operator can place the machine on a “delay” (where modelling of the machine's behaviour ceases) while the machine is undertaking non-productive operations. Once the machine has completed non-productive operations the delay can be removed and modelling of the machine's state recommenced. For example, after dumping a haulage machine may need to be refuelled. In this case the operator is notified (or observes) that the machine is performing a non-productive operation and places a delay on the machine to cease modelling. When the operator is informed (or observes) that the machine has refuelled and is returning to a productive operation state (e.g. travelling empty) the operator removes the delay and modelling of the machine's state recommences.
Alternatively, non-productive states may be handled automatically. This may be done, for example, by using the model to identify discrepancies in the machine's behaviour (i.e. significant departures from the expected machine state/state transitions) and using those discrepancies automatically identify that the machine has commenced non-productive operations. The discrepancies may be used with additional information to assist in this determination. For example, location information may indicate that the machine is attending a refuelling location, a maintenance location, or a shift-change location (indicating, respectively, that the machine is refuelling, undergoing maintenance, or changing driver). Fuel level information may indicate that the machine is running out of fuel, in turn indicating that a refuelling operation may be in progress.
As noted above, accurate information as to the particular operational state of a specific machine 102 at a given time can be analysed to generate information on the performance of the worksite 100 and/or the performance of individual worksite operators/machines.
For example, in the haulage machine material transportation example described above, accurate operational state information on the individual haulage machines 102b involved may be useful to assess one or more of the following:
The processes described above can, of course, be applied to worksite operations involving alternative operational states, state transitions, and observations.
For example, the process may be for a short-haul operation involving the transfer of material from a stockpile to a crusher. In this case: the relevant operation states may be identified as loading, hauling, and dumping. Other operations may be related to dozer, drill, dragline, water truck activities and cycles. For each different operation the relevant states, state transitions and observations are selected and an appropriate HMM generated, trained, tested and used to estimate the operational states of machines using the processes described above.
Disclosed herein is a computer-implemented method for estimating an actual operational state of a machine of interest at a point in time, the machine of interest being involved in a worksite operation comprising a plurality of different operational states, wherein the method comprises operating a computer processing unit to: receive a plurality of messages, each message comprising data relating to the machine of interest as it performs the worksite operation, the plurality of messages comprising data from at least two separate sources; process the data from the plurality of messages using a statistical process to generate an estimated actual operational state of the machine of interest at the point in time. The at least two separate data sources may comprise a first data source comprising data derived from a first sensor carried by the machine of interest and a second data source comprising data derived from a second sensor carried by the machine of interest. The at least two separate data sources comprise a first data source comprising data derived from the machine of interest and a second data source comprising data derived another worksite machine. The at least two separate data sources comprise a data source of contextual information. The plurality of messages may comprise at least one perceived state message comprising data relating to a perceived operational state of the machine of interest at a point in time. The source of the at least one perceived state message may be the machine of interest. The computer-implemented method may further comprise operating the computer processing unit to: process the estimated actual operational state against alert criteria; and responsive to the alert criteria being satisfied, raise an alert comprising at the estimated actual operational state of the machine of interest. The computer-implemented method may further comprise operating the computer processing unit to: process the plurality of operation messages using the statistically based process to generate uncertainty information associated with the estimated actual operational state of the machine of interest at the point in time, the uncertainty information defining an uncertainty with respect to the estimated actual operational state of the machine of interest at the point in time, and wherein the uncertainty information associated with the estimated actual operational state is used in the processing of the estimated actual operational state against the alert criteria. The computer processing unit may be operated to: receive the plurality of messages in real time; and process the plurality of messages in real time such that the estimated actual operational state of the machine of interest is generated in real time. The computer implemented method may further comprise operating the computer processing unit to: generate a computational model of the worksite operation, the computational model comprising the plurality of different operational states and plurality of observations relevant to the plurality of operational states, wherein each of the plurality of messages comprises data relating to an observation, and wherein processing the data from the plurality of messages using the statistical process comprises processing the data using the computational model to estimate the actual operational state of the machine of interest. The computer-implemented method may further comprise operating the computer processing unit to train the computational model using a training dataset. The computational model may comprise a Hidden Markov Model and the operational states may be hidden states of the Hidden Markov Model. Processing the data from the plurality of messages using the statistical process may comprise using a Viterbi Algorithm to process the data using the computational model to estimate the actual operational state of the machine of interest. The worksite process may comprise operating haulage machines to haul material from a loading location to a dumping location, and the machine of interest may a particular haulage machine. The plurality of different operational states may comprise: the machine of interest queuing at a loading location; the machine of interest spotting; the machine of interest loading; the machine of interest travelling full; the machine of interest queuing at a dumping location; the machine of interest dumping; the machine of interest travelling empty.
It will be understood that the invention disclosed and defined in this specification extends to all alternative combinations of two or more of the individual features mentioned or evident from the text or drawings. All of these different combinations constitute various alternative aspects of the invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2014203829 | Jul 2014 | AU | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/AU2015/050391 | 7/10/2015 | WO | 00 |
