Patent Application
20250118057
The present invention relates to classification (or regression) of data within data sets. In particular, this invention relates to assigning tags to data within one or more data sets to enhance the application of machine learning techniques to the one or more data sets. This invention also relates to a method of computer-aided quality control during data classification (or regression), as well as to a method of semi-automated tagging of data within one or more data sets.
In the application of supervised learning algorithms for classification (or regression) or regression, initially the training data needs to be labelled correctly, i.e. requires a dependent variable to be correctly assigned to each data point of the training data. A supervised learning algorithm is a regression or classification technique where the value for a dependent variable is known and assumed to be correct. The dependent variable is the variable that is being learned, which is discrete in the classification case and continuous in the regression case, and is also known as the tag or label in classification. The values of the dependent variable for the training data may have been obtained by manual annotation from a knowledgeable human expressing his/her opinion about what the ground truth value of the dependent variable would be, or by the ground truth value itself, obtained as a recording of the ground truth outcome by other means. For example in a geology application, the training set might be a set of 3D seismic scans, a datapoint would be a voxel in a scan, the dependent variable would be an indicator for resource endowment at the point in space represented by the voxel, and this value could have been discovered by drilling or sensing. In a legal application, the training set might a set of historical litigation cases, a datapoint would be a collection of documents that represents a litigation case, the ground truth value for the dependent variable would be the actual financial outcome of the case to the defendant. The fully labelled data is then used to train one or more supervised learning algorithms.
In many examples it is necessary to produce training data by a knowledgeable human adding tags to individual data points. Preparing this training data (i.e. classifying the data correctly) can be very labour-intensive, expensive and inconvenient, especially if a large amount of training data is to be used and if the quality of the data pre-preparation is not consistently high. Conventional interactive labelling can be computationally expensive and fail to deliver good results.
In conventional image analysis for auto insurance claims triage and repair estimates, images are captured in a controlled environment under standardised conditions (such as lighting, angle, zoom, background). To provide imagery from a controlled environment, special equipment is required at dedicated sites, and cars to be assessed are transported to those dedicated sites. This can be very expensive and inconvenient.
Aspects and/or embodiments can provide a method and/or system for labelling data within one or more data sets that can enable labelling of the one or more data sets with improved efficiency.
Further, aspects and/or embodiments can provide an improved system for image analysis for auto insurance claims triage and repair estimates which can alleviate at least some of the above problems. In particular the system can accommodate imagery from commodity hardware in uncontrolled environments.
According to one aspect, there is provided a method of modelling an unlabelled or partially labelled target dataset with a machine learning model for classification (or regression) comprising: processing the target dataset by the machine learning model; preparing a subgroup of the target dataset for presentation to a user for labelling or label verification; receiving label verification or user re-labelling or user labelling of the subgroup; and re-processing the updated target dataset by the machine learning model.
User labelling or label verification combined with modelling an unclassified or partially classified target dataset with a machine learning model can enable efficient labelling of an unlabelled component of the target dataset. By using a machine learning model for the modelling, images with a variety of imaging conditions (such as lighting, angle, zoom, background, occlusion) can be processed effectively. The machine learning algorithm may for example be a convolutional neural network, a support vector machine, a random forest or a neural network. Optionally the machine learning model is one that is well suited to performing classification or regression over high dimensional images (e.g. 10′000 pixels or more).
Optionally, the method may comprise determining a targeted subgroup of the target dataset for targeted presentation to a user for labelling and label verification of that targeted subgroup. This can enable a user to passively respond to queries put forward to the user, and so can lower the dependence on user initiative, skill and knowledge to improve the model and dataset quality.
Optionally, the preparing may comprise determining a plurality of representative data instances and preparing a cluster plot of only those representative data instances for presenting that cluster plot. This can reduce computational load and enable rapid preparation of a cluster plot for rapid display and hence visualisation of a high dimensional dataset. Optionally the plurality of representative data instances may be determined in feature space. Optionally the plurality of representative data instances may be determined in input space. Optionally the plurality of representative data instances may be determined by sampling. Optionally the preparing may comprise a dimensionality reduction of the plurality of representative data instances to 2 or 3 dimensions. Optionally the dimensionality reduction may be by t-distributed stochastic neighbour embedding.
Optionally, the preparing may comprise preparing a plurality of images in a grid for presenting that grid. Presentation in a grid can enable particularly efficient identification of images that are irregular.
Optionally, the preparing may comprise identifying similar data instances to one or more selected data instance by a Bayesian sets method for presenting those similar data instances. A Bayesian sets method can enable particularly efficient processing, which can reduce the time required to perform the processing.
According to another aspect, there is provided a method of producing a computational model for estimating vehicle damage repair with a convolutional neural network comprising: receiving a plurality of unlabelled vehicle images; processing the vehicle images by the convolutional neural network; preparing a subgroup of the vehicle images for presentation to a user for labelling or label verification; receiving label verification or user re-labelling or user labelling of the subgroup; and re-processing the plurality of vehicle images by the convolutional neural network.
User labelling or label verification combined with modelling target dataset that includes unlabelled images with a convolutional neural network can enable efficient classification (or regression) of unlabelled images of the target dataset. By using a convolutional neural network for the modelling, images with a variety of imaging conditions (such as lighting, angle, zoom, background, occlusion) can be processed effectively. Another machine learning algorithm may take the place of the convolutional neural network.
Optionally, the method may comprise determining a targeted subgroup of the vehicle images for targeted presentation to a user for labelling and label verification of that targeted subgroup. This can enable a user to passively respond to queries put forward to the user, and so can lower the dependence on user initiative, skill and knowledge to improve the model and dataset quality. Optionally, the preparing may comprise one or more of the steps for preparing data as described above.
Optionally, the method may further comprise: receiving a plurality of non-vehicle images with the plurality of unlabelled vehicle images; processing the non-vehicle images with the vehicle images by the convolutional neural network; preparing the non-vehicle images for presentation to a user for verification; receiving verification of the non-vehicle images; and removing the non-vehicle images to produce a plurality of unlabelled vehicle images. This can enable improvement of a dataset that includes irrelevant images.
The subgroup of vehicle images may all show a specific vehicle part. This can enable tagging of images by vehicle part. An image may have more than one vehicle part tag associated with it. The subgroup of vehicle images may all show a specific vehicle part in a damaged condition. This can enable labelling of images by damage status. The subgroup of vehicle images may all show a specific vehicle part in a damaged condition capable of repair. The subgroup of vehicle images may all show a specific vehicle part in a damaged condition suitable for replacement. This can enable labelling of images with an indication of whether repair or replacement is most appropriate.
According to another aspect, there is provided a computational model for estimating vehicle damage repair produced by a method as described above. This can enable generating a model that can model vehicle damage and the appropriate repair/replace response particularly well.
The computational model may be adapted to compute a repair cost estimate by: identifying from an image one or more damaged parts; identifying whether the damaged part is capable of repair or suitable for replacement; and calculating a repair cost estimate for the vehicle damage. This can enable quick processing of an insurance claim in relation to vehicle damage.
Optionally, to enhance usefulness, the computational model may be adapted to compute a certainty of the repair cost estimate. Optionally, to enhance usefulness, the computational model may be adapted to determine a write-off recommendation. Optionally, to enhance the quality of a repair cost estimate, the computational model may be adapted to compute its output conditional on a plurality of images of a damaged vehicle for estimating vehicle damage repair. Optionally, to enhance the quality of a repair cost estimate, the computational model may be adapted to receive a plurality of images of a damaged vehicle for estimating vehicle damage repair. Optionally, to enhance usefulness, the computational model may be adapted to compute an estimate for internal damage. Optionally, to enhance usefulness, the computational model may be adapted to request one or more further images from a user.
According to another aspect, there is provided software adapted to produce a computational model as described above. According to another aspect, there is provided a processor adapted to produce a computational model as described above.
Aspects and/or embodiments can extend to a method of modelling data substantially as herein described and/or as illustrated with reference to the accompanying figures.
Aspects and/or embodiments can also extend to a method of producing a computational model for estimating vehicle damage repair substantially as herein described and/or as illustrated with reference to the accompanying figures.
Aspects and/or embodiments can also extend to a computational model substantially as herein described and/or as illustrated with reference to the accompanying figures.
Aspects and/or embodiments can also extend to software for modelling data substantially as herein described and/or as illustrated with reference to the accompanying figures.
Aspects and/or embodiments can also extend to a system for modelling data substantially as herein described and/or as illustrated with reference to the accompanying figures.
Aspects and/or embodiments can also extend to methods and/or apparatus substantially as herein described with reference to the accompanying drawings.
Aspects and/or embodiments can also provide a computer program and a computer program product for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein, and a computer readable medium having stored thereon a program for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein.
Aspects and/or embodiments can also provide a signal embodying a computer program for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein, a method of transmitting such a signal, and a computer product having an operating system which supports a computer program for carrying out any of the methods described herein and/or for embodying any of the apparatus features described herein.
Any apparatus feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure, such as a suitably programmed processor and associated memory.
Any feature in one aspect may be applied to other aspects, in any appropriate combination. In particular, method aspects may be applied to apparatus aspects, and vice versa. Furthermore, any, some and/or all features in one aspect can be applied to any, some and/or all features in any other aspect, in any appropriate combination.
It should also be appreciated that particular combinations of the various features described and defined in any aspects can be implemented and/or supplied and/or used independently.
Furthermore, features implemented in hardware may generally be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly.
These and other aspects of the present invention will become apparent from the following exemplary embodiments that are described with reference to the following figures having like-reference numerals in which:
For approximately a decade, vehicle body shops and loss adjustors in numerous countries have been capturing photos of damaged vehicles as evidence to back repair estimates submitted to insurers or solicitors. With approximately 19 million motor claims in the US alone per year, and approximately 10 images per claim, a large body of imagery data for damaged vehicles exists.
Machine learning is an attractive tool for taking advantage of the existing vehicle damage imagery, and deep learning (and in particular convolutional neural networks) has made huge strides towards the automated recognition and understanding of high-dimensional sensory data. One of the fundamental ideas underpinning these techniques is that the algorithm can determine how to best represent the data by learning to extract the most useful features. If the extracted features are good enough (discriminative enough), then any basic machine learning algorithm can be applied to them to obtain excellent results. Convolutional neural networks (also referred to as convnets or CNNs) are particularly well suited to categorise imagery data, and graphic processor unit (GPU) implementations of convolutional neural networks trained by supervised learning have demonstrated high image classification (or regression) performance on ‘natural’ imagery (taken under non-standardised conditions and having variability in e.g. lighting, angle, zoom, background, occlusion and design across car models, including errors and irrelevant images, having variability regarding quality and reliability).
To take advantage of the large body of vehicle damage imagery for training a convolutional neural network the data needs to be as error-free as possible, and in particular images need to be labelled correctly. Industrial datasets pose novel problems to deep learning, such as dealing with noisy/missing/inconsistently or partially labelled data which may also include irrelevant data.
In order for the machine learning to perform good quality classification (or regression) it is necessary to ensure good data quality for training, and to train a sufficiently good model on the data. Conventionally a user is required to first prepare data for training by going through the data and (re-) labelling the data until satisfied with the quality. Then a model is trained on the cleaned data.
Labelling (and more generally cleaning) the training data set by virtue of a user assigning labels to an image is a very lengthy and expensive procedure to the extent of being prohibitive for commercial applications.
Significantly improved efficiency can be achieved if the preparation of the training data set and the training of the model are interleaved. This is not an intuitive approach as the algorithm starts learning with a dataset that is known to be deficient. It can however be very efficient as it takes advantage of the ability of machine learning algorithms to identify datasets that are dissimilar and potentially erroneous. Each iteration of model training informs the best approach for the subsequent relabelling iteration (and vice versa). The end result of this iterative process is a dataset of sufficient quality and a model providing sufficiently discriminative features on this dataset.
The data may can be in the form of images (with each image representing an individual dataset), or it can be any high-dimensional data such as text (with each word for example representing an individual dataset) or sound.
In order to enable use of existing imagery data for training a convolutional neural network semi-automatic labelling is now described.
Semi-automatic labelling semi-automates the labelling of datasets. A model is trained on data that is known to include errors. The model attempts to model and classify (or regress) the data. The classification, also referred to as the labelling or the tagging, of selected data points (individual images or groups of images) are reviewed by a user (also referred to as an oracle or a supervisor) and corrected or confirmed. Labels are iteratively refined and then the model is refined based on the labelled data. The user can proactively review the model output and search for image for review and labelling, or the user can passively respond to queries from the model regarding labelling of particular images.
A sequence of operations for semi-automatic labelling with proactive user review is:
In an example of a semi-automatic labelling procedure as set out above approximately 30'000 images can be labelled in an hour with a single user into a scheme with 18 classes with 90% accuracy.
In the case of passive user response to queries (also referred to as targeted supervision), Steps 3 and 4 of the sequence described above are as follows:
Passive and proactive user review can also be combined by providing both alongside one another.
Step 3c ‘assign labels to some/all feature points’ can be performed for classification by a clustering technique such as partitioning the feature space into class regions. Step 3c can also be performed for regression by a discretising technique such as defining discrete random values over the feature space.
As part of Step 6 (fine tuning) following additional steps may be executed:
There are a number of further considerations to take into account in implementing the sequence set out above, including:
Some techniques that can be used to implement the semi-automatic labelling described above are:
A pre-trained convolutional neural network may for example be trained on images from the ImageNet collection.
A similarity search may be executed in order to find images that are similar to a particular image or group of images of interest. This can enable a user to find an individual image of particular interest (for example an image of a windscreen with a chip in a cluster of windscreen images), find further images that are similar, and to provide a label to the images collectively.
Now a method of performing dimensionality reduction on the feature set (Step 3.c above) is described in more detail. In an example the feature set is a 4096-dimensional vector (and more generally an N-dimensional vector) having values in the range of approximately −2 to 2 (and more generally in a typical range). Dimension reduction to two or three dimensions (as can be intuitively understood by a human) can require considerable computational resources and take significant time. In order to shorten this computationally labour-intensive step, the data set is clustered in feature space and from each cluster a single representative data instance (also referred to as a centroid; a k-means cluster centroid for example) is selected for further processing. The dimension reduction is then performed on the representative data only, thereby reducing the computational load to such an extent that very rapid visualisation of very large data sets is possible. Data-points from the dataset are not individually shown in the cluster plot to the user, however the diameter of a circle in the cluster plot shown to the user indicates the number of data-points that are near the relevant representative data instance in feature-space, and hence presumed to have identical or similar label values. By selection of a circle in the cluster plot the user is presented with all of the images represented by that circle. This allows a user to check all the images represented by the representative. The scaling of the circles can be optimised and/or adjusted by a user for clarity of the display.
Now a method of performing a similarity search is described in more detail. The images are represented in feature-space by high-dimensional vectors (such as 4096-dimensional vectors), having a range of values (such as approximately from −2 to 2). Performing a similarity search on a large number of such vectors can be computationally labour-intensive and take significant time. Bayesian sets can provide a very quick and simple means of identifying similar entities to an image or group of images of particular interest. In order to apply a Bayesian set method the data (here the high-dimensional vectors) is required to be binary rather than having a range of values. In order to apply a Bayesian set method the feature set vectors are converted into binary vectors: values that are near-zero are changed to zero, and the values that are farther away from zero are changed to one. For similarity searching by the Bayesian set method this can produce good results. The application of Bayesian sets to convolutional neural networks (or more generally machine learning models suitable for images and with sparse representations) is particularly favourable as convolutional neural networks typically produce feature sets with sparse representations (lots of zeros in the vector) which are consequently straightforward to cast to binary vectors with sparse representations in the context of semi auto labelling.
Now semi-automatic labelling applied to vehicle damage estimation is described in more detail. For a given instance of vehicle damage the outcome is a prediction of the repairs that are necessary and an estimate of the corresponding repair cost based on natural images of the damaged vehicle. This can enable an insurer for example to make a decision as to how to proceed in response to the vehicle damage. The outcome may include a triage recommendation such as ‘write the vehicle off, ‘significant repairs necessary’, or ‘light repairs necessary”.
In order to produce a repair estimate, the procedure is broken down as follows for optimal processing:
By this procedure a repair estimate can be produced at first notice of loss, from images captured by a policyholder for example with a smartphone. This can enable settling of a claim almost immediately after incurrence of damage to a vehicle. It can also enable rapid selection, for example via mobile app, of:
Images can be supplied for a repair estimate at a time point later than the first notice of loss, for example after official services such as police or first aiders have departed or at a vehicle body shop or other specialised centre. An output posterior distribution of the repair estimate can be produced to provide more insight e.g. 95% confidence interval for a repair estimate; or a probability of write off. The repair estimate process can be dual machine/human generated, for example by passing the estimation over to a human operator if the estimate given by the model only has low confidence or in delicate cases. Parties other than the policyholder can capture images (e.g. a co-passenger in the damaged vehicle, another person involved in the accident, police, ambulance/first aid staff, loss adjustor/assessor, insurer representative, broker, solicitor, repair workshop personnel). The image(s) provided for the repair estimate may be from a camera or other photographic device. Other related information can be provided to the policyholder such as an excess value and/or an expected premium increase to dis-incentivise claiming.
By implementing repair estimation as described here both an insurer and a policyholder can enjoy a number of benefits. For example, an insurer can:
The policyholder can enjoy superior customer service and take advantage of suppliers bidding for custom. Certain part suppliers can benefit from preferred supplier status. Vehicle repairers and bodyshops can avoid spending time preparing estimates.
In the steps described above a convolutional neural network is employed. A multi-instance learning (MIL) convolutional neural network that can accommodate multi-image queries may perform substantially better than a convolutional neural network for single-image queries. Multiple images can in particular help to remove imagery noise from angle, lighting, occlusion, lack of context, insufficient resolution etc. In the classification case, this distinguishes itself from traditional image classification, where a class is output conditional on a single image. In the context of collision repair estimating, it may often be impossible to capture, in a single image, all the information required to output a repair estimate component. In an example the fact that a rear bumper requires repair can only be recognised by capturing a close-up image of the damage, which loses the contextual information that is required to ascertain that a part of the rear bumper is being photographed. By training a machine learning model that uses the information in multiple images in the example the machine learning model can output that the rear bumper is in need of repair. In a convolutional neural network architecture that can accommodate multi-image queries a layer is provided in the convolutional neural network that pools across images. Maximum pooling, average pooing, intermediate pooling or learned pooling can be applied. Single image convolutional neural networks may be employed for greater simplicity.
Now a procedure for producing a model that can accomplish Steps 1 and 2 of producing a repair estimate as describe above—recognising a set of damaged parts and predicting a ‘repair’/‘replace’ label—is described in more detail. This is achieved essentially by solving labelling problems with semi-automatic labelling as described above. This procedure is applied to a dataset that includes unlabelled vehicle images for every vehicle part that is to be recognised/diagnosed.
Now an adaptation for internal damage prediction is described in more detail. Internal damage prediction can be implemented for example with predictive analytics such as regression models. Images of a damaged vehicle do not permit direct observation of internal parts.
In the process described above the sequence of the steps can be varied. More information is available in an image of a damaged part than in a binary repair/replace decision. Hence by regressing the repair costs to images the accuracy can be improved as compared to an image-less model.
An implementation of the repair estimate may include further features such as:
It will be understood that the present invention has been described above purely by way of example, and modifications of detail can be made within the scope of the invention.
Each feature disclosed in the description, and (where appropriate) the claims and drawings may be provided independently or in any appropriate combination.
Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15765275 | Apr 2018 | US |
| Child | 18658748 | US |
