Patent Application
20230131444
This invention relates generally to detecting fraudulent activity at self-checkout terminals of retail stores, and in particular, to systems and methods for detecting “ticket switching,” a fraudulent practice of scanning a barcode of a cheaper product when attempting to actually purchase a more expensive product at a self-checkout terminal.
Retailers lose millions, if not billions, of dollars annually due to fraudulent activities at their self-checkout terminals. One such fraudulent scanning practice, which involves scanning a barcode of a cheaper item when actually attempting to check out a more expensive item at the self-checkout terminal, is called ticket-switching. As a result of ticket-switching, since the item is placed in the product-scanning area of the self-checkout terminal such that the bar code of a less expensive item (not the actual barcode of the more expensive item) is scanned instead, the person engaged in the fraudulent activity is able to buy a more expensive item (e.g., a video game, electronic device, home improvement tool, etc.) at a cheaper price than the actual price for this item. Given that large retailers have millions of products passing through their self-checkout terminals daily, ticket switching results in significant profit losses for retailers, and it would be desirable for retailers to curb the occurrence of ticket-switching to a minimum.
Disclosed herein are embodiments of systems and methods of detecting fraudulent activity at a self-checkout terminal of a retail store. This description includes drawings, wherein:
Elements in the figures are illustrated for simplicity and clarity and have not been drawn to scale. For example, the dimensions and/or relative positioning of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of various embodiments of the present invention. Also, common but well-understood elements that are useful or necessary in a commercially feasible embodiment are often not depicted in order to facilitate a less obstructed view of these various embodiments of the present invention. Certain actions and/or steps may be described or depicted in a particular order of occurrence while those skilled in the art will understand that such specificity with respect to sequence is not actually required. The terms and expressions used herein have the ordinary technical meaning as is accorded to such terms and expressions by persons skilled in the technical field as set forth above except where different specific meanings have otherwise been set forth herein.
The following description is not to be taken in a limiting sense, but is made merely for the purpose of describing the general principles of exemplary embodiments. Reference throughout this specification to “one embodiment,” “an embodiment,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.
Generally, methods and systems for detecting fraudulent activity at self-checkout terminals of a retail store include a scanner for scanning an identifier of a candidate product located in the product-scanning area of the self-checkout terminal and one or more sensors that detect one or more physical characteristics of the candidate product located in the product-scanning area of the self-checkout terminal. A computing device then correlates the obtained electronic data corresponding to actual identifying characteristic information associated with the candidate product to the reference physical characteristic information associated with the reference product in order to generate a similarity score between the actual and reference physical characteristic information. If the similarity score is above a predetermined similarity threshold, the self-checkout terminal is permitted to process a purchase of the candidate product, but if the similarity score is below the predetermined similarity threshold, the self-checkout terminal is restricted from processing the purchase.
In some embodiments, a system for detecting fraudulent activity at a self-checkout terminal of a retail store includes a scanner located proximate a product-scanning area of the self-checkout terminal and configured to scan an identifier of a candidate product located in the product-scanning area of the self-checkout terminal, as well as at least a first sensor located proximate the product-scanning area of the self-checkout terminal and configured to detect at least one physical characteristic of the candidate product located in the product-scanning area of the self-checkout terminal. The system further includes an electronic database configured to store at least one of: electronic data corresponding to the identifier of the candidate product; and electronic data corresponding to reference physical characteristic information associated with a reference model of the candidate product, the reference physical characteristic information associated with at least one image of a reference product, identical to the candidate product, captured by the at least one sensor during a scan of the reference product by the scanner. The system further includes a processor-based computing device in communication with the at least one sensor and the electronic database, with the computing device being configured to: obtain electronic data corresponding to actual identifying characteristic information associated with a candidate product, the actual identifying characteristic information associated with at least one image of the candidate product captured by the at least one sensor at a time of a scan of the candidate product by the scanner; correlate the electronic data corresponding to the actual identifying characteristic information associated with the candidate product obtained by the at least one sensor during the scan of the candidate product by the scanner to the electronic data corresponding to the reference physical characteristic information associated with the reference product in order to generate a similarity score between the actual identifying characteristic information and the reference physical characteristic information; determine if a correlation of the actual identifying characteristic information associated with the candidate product to the reference physical characteristic information associated with the reference product indicates whether the similarity score between the actual identifying characteristic information associated with the candidate product and the reference physical characteristic information associated with the reference product is above or below a predetermined similarity threshold. If the similarity score is above the predetermined similarity threshold, the computing device permits the self-checkout terminal to process a purchase of the candidate product. If similarity score is below the predetermined similarity threshold, the computing device restricts the self-checkout terminal from processing the purchase of the candidate product.
In some embodiments, a method of detecting fraudulent activity at a self-checkout terminal of a retail store includes: scanning, via a scanner located proximate a product-scanning area of the self-checkout terminal, an identifier of a candidate product located in the product-scanning area of the self-checkout terminal; detecting, via a first sensor located proximate the product-scanning area of the self-checkout terminal, at least one physical characteristic of the candidate product located in the product-scanning area of the self-checkout terminal; providing an electronic database configured to store at least one of: electronic data corresponding to the identifier of the candidate product; and electronic data corresponding to reference physical characteristic information associated with a reference model of the candidate product, the reference physical characteristic information associated with at least one image of a reference product, identical to the candidate product, captured by the at least one sensor during a scan of the reference product by the scanner; and providing a processor-based computing device in communication with the at least one sensor and the electronic database; obtaining, by the computing device, electronic data corresponding to actual identifying characteristic information associated with a candidate product, the actual identifying characteristic information associated with at least one image of the candidate product captured by the at least one sensor at a time of a scan of the candidate product by the scanner; correlating, by the computing device, the electronic data corresponding to the actual identifying characteristic information associated with the candidate product obtained by the at least one sensor during the scan of the candidate product by the scanner to the electronic data corresponding to the reference physical characteristic information associated with the reference product in order to generate a similarity score between the actual identifying characteristic information and the reference physical characteristic information; determining, via the computing device, if a correlation of the actual identifying characteristic information associated with the candidate product to the reference physical characteristic information associated with the reference product indicates whether the similarity score between the actual identifying characteristic information associated with the candidate product and the reference physical characteristic information associated with the reference product is above or below a predetermined similarity threshold; permitting the self-checkout terminal to process a purchase of the candidate product in response to a determination by the computing device that the similarity score is above the predetermined similarity threshold; and restricting the self-checkout terminal from processing the purchase of the candidate product in response to a determination by the computing device that the similarity score is below the predetermined similarity threshold.
In some embodiments, the self-checkout terminal 110 is configured to require a customer to physically grasp the product 190 and bring the product 190 into the product-scanning area 115 of the self-checkout terminal 110 to permit the scanner 120 to scan the product 190 in order to obtain identifying information associated with the product 190. In other embodiments, the self-checkout terminal 110 is configured to include a conveyor belt-like product advancement surface that moves the 190 in proximity to the scanner 120, such that the scanner 120 is permitted to scan the product 190 in order to obtain identifying information associated with the product 190.
The scanner 120 of the exemplary system 100 depicted in
In the embodiment shown in
In some aspects, the first sensor 130 that detects at least one physical characteristic of the candidate product 190 located in the product-scanning area 115 of the self-checkout terminal 110 is in the form of a camera that provides for at least one of: image analysis of the product 190 (e.g., size, shape, and color of one or more sides of the product), text recognition on the candidate product 190 (e.g., product name), and/or pattern recognition on the product 190 (e.g., color pattern or barcode pattern). In certain aspects, the sensor 130 is a “top-down” camera positioned such that it is located above the candidate product 190 being scanned by the scanner 120 in the product-scanning area 115. In one aspect, the camera sensor 130 generates video image data that includes a timestamp attached to each frame.
In some embodiments, as will be described in more detail below, the physical characteristic information of a candidate product 190 detected by the sensor 130 is transmitted by the sensor 130 and later correlated by the computing device 150 to the reference physical characteristics associated with the candidate product 190 (which information may be obtained by the computing device 150 from the electronic database 170). In certain aspects, if this correlation comparison of the actual physical characteristics of the candidate product 190 that is captured by the sensor 130 to the reference physical characteristics associated in the electronic database 140 with the reference product 190 indicates a discrepancy that exceeds a certain tolerated threshold between the candidate product 190 and the reference product data (e.g., a pre-defined difference between the similarly scores of the candidate product 190 and the reference product data), an alert (which may be indicative of ticket-switching) is triggered, and the self-checkout terminal 110 becomes restricted from processing the purchase of the candidate product 190.
In certain implementations, the camera-based sensor 130 is configured to capture an image of the candidate product 190 located in the product-scanning area 115 of the self-checkout terminal 110, and to compress the captured image prior to transmitting the compressed image to another electronic device (e.g., electronic database 140, computing device 150, etc., which will be discussed in more detail below). This image compression by the camera-based sensor 130 advantageously reduces the storage requirements of the electronic database 140 (as compared to capturing and transmitting full-size images), and advantageously reduces the processing power required of the computing device 150 to process the compressed image (as compared to the full-size image) when attempting to detect any physical characteristic of the candidate product 190 based on processing the image captured by the camera-based sensor 130 and stored in the electronic database 140.
In one aspect, the self-checkout terminal 110 may include a weight sensor 135 in the form of a plate or the like positioned in the product-scanning area 115 of the self-checkout terminal 110 that provides for the measurement of the weight of the product 190 when the product 190 is located in the product-scanning area 115. In some embodiments, as will be described in more detail below, a weight of a candidate product 190 detected by the weight sensor 135 is transmitted by the weight sensor 135 (e.g., to the electronic database 140 or to the computing device 150) and later correlated by the computing device 150 to a reference weight of the product 190 (which predefined weight may be obtained by the computing device 150 from the electronic database 170). In certain aspects, if this correlation comparison, by the computing device 150, of the actual weight of the candidate product 190 that is measured by the weight sensor 135 to the reference weight associated with the candidate product 190 indicates a discrepancy that exceeds a certain tolerated threshold (e.g., a pre-defined difference between the between the two weights), an alert (which may be indicative of ticket-switching) is triggered, and the self-checkout terminal 110 becomes restricted (e.g., in response to an alert signal received from the computing device 150) from processing the purchase of the candidate product 190.
In some aspects, a product physical characteristic-detecting sensor 130 may be an infrared and/or a thermal sensor that measures the temperature of the candidate product 190 and/or insulation integrity of the packaging of the product 190 when the candidate product 190 is located in the product-scanning area 115. In some embodiments, the product physical characteristic-detecting sensor 130 is a 3D scanner or sensor configured to detect and/or measure the shapes and/or dimensions of the candidate product 190 on one or more sides when the candidate product 190 is located in the product-scanning area 115.
It will be appreciated that the physical locations of the scanner 120 and sensor 130 relative to the product-scanning area 115 have been shown by way of example only, and that in some embodiments, the scanner 120 and the sensor 130 (which may be one sensor or an array of various sensors), may be positioned in various locations and orientations relative to the product-scanning area 115. It will also be appreciated that one or more of the sensors 130 and 135 can be physically incorporated (e.g., coupled to, embedded, etc.) into the physical structure of the self-checkout terminal 110 and/or the computing device 150, or may be stand-alone sensors.
In the embodiment shown in
In some embodiments, the electronic database 140 may be stored, for example, on non-volatile storage media (e.g., a hard drive, flash drive, or removable optical disk) internal or external to the computing device 150 and/or the self-checkout terminal 110, or internal or external to computing devices distinct from the computing device 150 and the self-checkout terminal 110. In some embodiments, the electronic database 140 may be cloud-based.
In some embodiments, the exemplary electronic database 140 of
The exemplary system 100 of
The computing device 150 may be a stationary or portable electronic device, for example, a desktop computer, a laptop computer, a tablet, a mobile phone, or any other electronic device including a processor-based control circuit (i.e., control unit). In the embodiment of
In some embodiments, the system 100 includes one or more localized Internet-of-Things (IoT) devices and controllers in communication with the computing device 150. As a result, in some embodiments, the localized IoT devices and controllers can perform most, if not all, of the computational load and associated monitoring that would otherwise be performed by the computing device 150, and then later asynchronous uploading of summary data can be performed by a designated one of the IoT devices to the computing device 150, or a server remote to the computing device 150. In this manner, the computational effort of the overall system 100 may be reduced significantly. For example, whenever a localized monitoring allows remote transmission, secondary utilization of controllers keeps securing data for other IoT devices and permits periodic asynchronous uploading of the summary data to the computing device 150 or a server remote to the computing device 150.
In addition, the periodic asynchronous uploading of summary data may include a key kernel index summary of the data as created under nominal conditions. In an exemplary embodiment, the kernel encodes relatively recently acquired intermittent data (“KRI”). As a result, in some embodiments, KRI includes a continuously utilized near term source of data, but KRI may be discarded depending upon the degree to which such KRI has any value based on local processing and evaluation of such KRI. In an exemplary embodiment, KRI may not even be utilized in any form if it is determined that KRI is transient and may be considered as signal noise. Furthermore, in some embodiments, the kernel rejects generic data (“KRG”) by filtering incoming raw data using a stochastic filter that provides a predictive model of one or more future states of the system and can thereby filter out data that is not consistent with the modeled future states which may, for example, reflect generic background data. In some aspects, KRG incrementally sequences all future undefined cached kernels of data in order to filter out data that may reflect generic background data. In other aspects, KRG incrementally sequences all future undefined cached kernels having encoded asynchronous data in order to filter out data that may reflect generic background data.
With reference to
The control circuit 210 can be configured (for example, by using corresponding programming stored in the memory 220 as will be well understood by those skilled in the art) to carry out one or more of the steps, actions, and/or functions described herein. In some embodiments, the memory 220 may be integral to the processor-based control circuit 210 or can be physically discrete (in whole or in part) from the control circuit 210 and is configured non-transitorily store the computer instructions that, when executed by the control circuit 210, cause the control circuit 210 to behave as described herein. (As used herein, this reference to “non-transitorily” will be understood to refer to a non-ephemeral state for the stored contents (and hence excludes when the stored contents merely constitute signals or waves) rather than volatility of the storage media itself and hence includes both non-volatile memory (such as read-only memory (ROM)) as well as volatile memory (such as an erasable programmable read-only memory (EPROM))). Accordingly, the memory and/or the control unit may be referred to as a non-transitory medium or non-transitory computer readable medium.
The control circuit 210 of the computing device 150 is also electrically coupled via a connection 235 to an input/output 240 that can receive signals from, for example, from the scanner 120, the sensors 130, 135, etc., the electronic database 170, and/or from another electronic device (e.g., a hand-held electronic device of a worker tasked with monitoring the self-checkout terminal 110) over the network 125. The input/output 240 of the computing device 150 can also send signals to the self-checkout terminal 110, for example, an alert signal that halts operation of the self-checkout terminal 110 and restricts the self-checkout terminal 110 from processing the purchase of the candidate product 190 until, for example, input from a worker tasked with watching the self-checkout terminal 110 indicates that the candidate product 190 has been verified.
The processor-based control circuit 210 of the computing device 150 shown in
For example, such manual control by an operator may be via the user interface 250 of the computing device 150, via another electronic device of the operator, or via another user interface and/or switch, and may include an option to override the alert initially generated by the computing device 150 upon a visual inspection of the self-checkout terminal 110 and the candidate product 190 by the worker, and the worker’s confirmation that no fraudulent activity has occurred with respect to the checkout of the candidate product 190 at the self-checkout terminal 110. In some embodiments, the user interface 250 of the computing device 150 may also include a speaker 280 that provides audible feedback (e.g., alerts) to the worker. It will be appreciated that the performance of such functions by the processor-based control circuit 210 of the computing device 150 is not dependent on a human operator, and that the control circuit 210 may be programmed to perform such functions without a human operator.
In some embodiments, the control circuit 210 of the computing device 150 is configured obtain the above-referenced electronic data from the electronic database 140, and to process such data in order to make a determination regarding the likelihood that the customer attempting to purchase the candidate product 190 via the self-checkout terminal 110 is engaging in fraudulent activity (e.g., ticket-switching). In some aspects, for example, the control circuit 210 of the computing device 150 is programmed to obtain electronic data corresponding to actual identifying characteristic information associated with a candidate product 190. As mentioned above, in certain embodiments, the actual identifying characteristic information represents electronic data associated with and/or based upon at least one image of the candidate product 190 that is captured by one or more sensors (e.g., the sensor 130) at a time of a scan by the scanner 120 of the candidate product 190 located in the product-scanning area 115.
In some embodiments, after obtaining (e.g., from the electronic database 140 or directly from the sensor(s) 130) the actual identifying characteristic information associated with the candidate product 190 being scanned by the scanner 120 in the product-scanning area 115 and obtaining (e.g., from the electronic database 140) the electronic data corresponding to the reference physical characteristic information associated with the reference product 190, the control circuit 210 of the computing device 150 is programmed to correlate the electronic data corresponding to the actual identifying characteristic information associated with the candidate product 190 to the electronic data corresponding to the reference physical characteristic information associated with the reference product 190 in order to generate a similarity score between the actual identifying characteristic information and the reference physical characteristic information. Generally, a correlation of the actual physical characteristic information associated with the candidate product 190 to the reference physical characteristic information associated with the reference product 190 requires the presence of corresponding reference physical characteristic information in the electronic database 140 (i.e., requires that a product identical to the candidate product 190 has previously been processed at the self-checkout terminal 110, and that a reference model has been generated and recorded in the electronic database 140 in association with that reference scan).
In some situations, the candidate product 190 may be a product that is being processed at the self-checkout terminal 110 for the first time, and the electronic database 140 may not yet have electronic data representing the reference model data indicative of the reference physical characteristics for the candidate product 190. Accordingly, in some aspects, the control circuit 210 of the computing device 150 is programmed to automatically generate a reference physical characteristics model for a first time-scan candidate product 190, and to update the electronic database 140 to include electronic data representing a reference model associated with this candidate product 190 (i.e., auto-enrollment) into the electronic database 140.
To that end, in one embodiment, in response to a determination by the computing device 150 that the electronic database 140 does not include a reference model for the candidate product 190, the computing device is configured to initiate a conversion of the electronic data (e.g., one or more digital images captured by a camera sensor 130) corresponding to the actual identifying characteristic information associated with a candidate product 190 into electronic data (e.g., a set of numerical values representing one or more fixed size vectors) corresponding to the reference model for the candidate product 190 for use in subsequent scans of other identical candidate products 190. In one implementation, after this conversion is complete, the computing device 150 transmits the electronic data corresponding to the generated reference model for the candidate product 190 to the electronic database 140 for storage. An exemplary logic flow relating to auto-enrollment is depicted in
With reference to
In certain aspects, the electronic data corresponding to the reference physical characteristic information associated with the reference model of each candidate product 190 is stored in the electronic database 140 in a form of a set of numerical values representing one or more fixed size vectors (e.g., 512-dim) associated with the reference product 190. In one aspect, the one or more fixed size vectors stored in the electronic database 140 as a reference model for a given product 190 represent an output of an encoding, via a neural network (e.g., a Visual Geometry Group (VGG) deep convolutional neural network, a Residual Neural Network (Resnet), Inception Neural Network, or the like), of one or more images of a reference product (acquired by a sensor 130) into the fixed size vectors associated with the reference product.
When a candidate product 190 is scanned by the scanner 120 and at least one image of the candidate product 190 is captured by the sensor 130, the electronic data representative of the image(s) may be transmitted by the sensor 130 over the network 125 to the electronic database 140 to be obtained by the computing device 150, or directly to the computing device 150. In one aspect, after the electronic data corresponding to actual identifying characteristic information associated with a candidate product 190 (e.g., digital image data captured by the sensor 130) is obtained by the computing device 150, the control circuit 210 of the computing device 150 is programmed to encode the image(s) of the candidate product 190 captured by the sensor(s) 130 at a time of a scan of the candidate product 190 by the scanner 120 into a set of numerical values representing at least one fixed size vector associated with the candidate product 190. After this encoding of image data into fixed size vector values is complete, the control circuit 210 of the computing device 150 is programmed to correlate aggregate average numerical values representing the fixed size vectors associated with the reference product 190 to the aggregate average numerical values representing the fixed size vectors associated with the candidate product 190. This correlation enables the control circuit 210 of the computing device 150 to determine (i.e., calculate) a similarity score indicative of how similar or dissimilar the numerical values representing the fixed size vectors associated with the candidate product 190 are to the numerical values representing the fixed size vectors associated with the reference product 190.
In some embodiments, the computing device 150 is programmed to monitor checkout events (e.g., a scan of an RFID or barcode 195 of the candidate product 190 by the scanner 120) that occur at the self-checkout terminal 110, while at the same time analyzing the video frames received from a camera sensor 130. An exemplary checkout event may be represented by electronic data including the unique identifier of the candidate product 190, as well as a timestamp of when the candidate product 190 was scanned. In one aspect, after a check-out event is received, the control circuit 210 of the computing device 150 is programmed to seek the video frames from a smaller time window (usually 1-2 seconds), which covers the detected check-out event, then analyze the video frames in an attempt to detect whether the candidate product 190 associated with the identifier detected by the scanner 120 during the attempted checkout event is actually present or not present in the product-scanning area 115 of the self-checkout terminal 110.
In some aspects, the control circuit 210 is programmed to, based on the correlation of the actual identifying characteristic information associated with the candidate product 190 to the reference physical characteristic information associated with the reference product 190, the control circuit 210 of the computing device 150 is programmed to cause the computing device 150 to transmit a signal to the electronic database 140 in order to update the electronic database 140 by replacing the reference physical characteristic information associated with the reference product 190 stored in the electronic database 140 with the actual identifying characteristic information associated with the candidate product 190. As described in more detail below, this update of the electronic database 140 generally occurs when the control circuit 210 of the computing device 150 determines that the actual identifying characteristic information associated with the candidate product 190 has a similarity score that warrants replacement of the reference physical characteristic information associated with the candidate product 190 with the actual identifying characteristic information associated with the candidate product 190, such that the actual identifying characteristic information for the candidate product 190 becomes the new reference model.
In various implementations, the control circuit 210 of the computing device 150 is programmed to continuously build and update the reference model data set associated with the products 190 being processed at the self-checkout terminal 110. As mentioned above, when a candidate product 190 comes through the self-checkout terminal 110 for the first time, the computing device 150 generates a reference model data set in association with the candidate product 190, and transmits this reference model data set to the electronic database 140 for storage and future retrieval (this process is referred to above as “enrollment”). In some aspects, as the computing device 150 continuously acquires additional electronic data representative of the scans of additional candidate products 190, the control circuit 210 of the computing device is programmed to process the obtained image data as discussed above, and to score the incoming data in order to determine whether the reference model data associated with a given scanned candidate product 190 should be preserved, or replaced with the newly received image data based on the current scan of the candidate product 190.
Generally, the control circuit 210 of the computing device 150 is programmed to process the reference product image-associated data and the candidate product image-associated data to output (using an encoding network) candidate fixed size vector data and reference fixed size vector data. In some embodiments, as discussed in more detail below, this candidate and reference vector data is then processed via a scoring neural network to generate a quality score for the images (which is then normalized to achieve a final similarity score), as well as via a verification neural network to generate a similarity score (which is then weighted and averaged to achieve the final similarity score).
For example, in certain aspects, the control circuit 210 of the computing device 150 is programmed to analyze, via a scoring neural network, the fixed size vector(s) associated with the candidate product 190, and to generate an image quality score indicative of an overall quality of the image(s) of the candidate product captured by the sensor(s) 130 at the time of the scan of the candidate product 190 by the scanner 120. In some aspects, the fixed size vector representation (e.g., 512-dim) captures the high-level visual features of the input image of the candidate product 190 that was captured by the sensor 130 during the scanning of the candidate product by the scanner 120, and the scoring neural network takes the vector representation and outputs a quality score from 0 to 1, indicating how informative the corresponding image is, with values close to 1 being considered highly informative, and values close to 0 being considered less informative.
For example, if the sensor 130 captures an image of the candidate product 190, where the product is occluded by an interfering object positioned between the sensor 130 and the candidate product 190, such an image is likely to get a quality score of close to 0, since it is hard to see the actual product from the image. When, based on this analysis, the control circuit 210 determines that the image quality score is above a predetermined image quality threshold, the control circuit 210 of the computing device 150 is programmed to correlate the aggregate average numerical values representing the fixed size vector(s) associated with the reference product to the aggregate average numerical values representing the fixed size vector(s) associated with the candidate product 190 in order to determine the similarity score.
In certain implementations, the computing device 150 is implemented to include a verification neural network, which takes a pair of vectors representing (1) an image (also referred to herein as a “query image”) of a candidate product 190 captured from the current scan of the candidate product 190 by the scanner 120 and sensor 130, and (2) a reference image associated with the candidate product 190 in the electronic database 140, and outputs a similarity score from 0 to 1, indicating how similar the pair of images are (with values close to 0 being less similar and values close to 1 being highly similar). For detection, the computing device 150 can be configured to take: (1) several (e.g., 1, 2, 3, or more) query images of the candidate product 190 captured by the sensor 130 when the candidate product 190 is located in the product-scanning area 115 within a time window of when a scan of the candidate product 190 by the scanner 120 is detected; and (2) several (e.g., 1, 2, 3, or more) reference images (also referred to herein as an “enrollment set”) stored in the electronic database 140 in association with the candidate product 190, and compute similarity scores for all of the possible pairs of the query images and the reference images, then average the computed similarity scores using the image quality scores as weights (e.g., by filtering out the contribution from lower-scoring, less informative images).
In some implementations, the control circuit 210 of the computing device 150 calculates and outputs a similarity score for every query-reference pair in a scan. That is, in some implementations, if there are 6 query images and 10 reference images, there are 60 pairs that each produce a confidence score (and every additional sensor would output a score for each of the query-reference data pairs). This analysis, which takes into account and analyzes several outputs from the scan-instance verification model, enables the control circuit 210 of the computing device 150 to make an aggregate decision at the scan level, not at the individual sensor-output level, with the aggregate decision providing a higher confidence in the determination of the similarity between the query images of the candidate product 190 captured by the sensor(s) 130 and the reference model images stored in the electronic database 140.
In certain aspects, if the aggregated score (ranging from 0 to 1), i.e., the normalized perceived distance (referred to herein as “NPD”), is more than (i.e., above) a fixed threshold, the system will trigger an alert for the self-checkout terminal 110. Thus, in some aspects, the control circuit 210 of the computing device 150 determines if a correlation of the actual identifying characteristic information associated with the candidate product 190 to the reference physical characteristic information associated with the reference product indicates whether the aggregated similarity score between the actual identifying characteristic information associated with the candidate product 190 and the reference physical characteristic information associated with the reference product is above or below the predetermined similarity threshold, and then: (1) if the similarity score is above the predetermined similarity threshold, the control circuit 210 of the computing device 150 is programmed to permit the self-checkout terminal 110 to process a purchase of the candidate product 190; and (2) if the similarity score is below the predetermined similarity threshold, the control circuit 210 of the computing device 150 is programmed to restrict the self-checkout terminal 110 from processing the purchase of the candidate product 190.
In particular, in some aspects, the control circuit 210 of the computing device 150 is programmed to compare the data generated in response to receipt of new scan data for a new candidate product 190 with the existing reference model set stored in the electronic database, and to take the candidate product 190 with the lowest average similarity score (compared to the reference model data set), and then compare it to the reference data point with the highest average similarity score (compared to others in reference model set), and to remove that reference data set having the highest average similarity score from the electronic database 140 and to replace this reference data set with the candidate data associated with the lowest average similarity score.
In some embodiments, the auto-updating of the reference model data stored in the electronic database 140 in association with the candidate products 190 is described below in more detail with reference to the logic flow 500 of
With reference to
As can be seen in
In Table 1 above, the image data acquired for candidate product C1 has the lowest average similarity score (i.e., 78.80) relative to the image data reference values R1, R2, R3, R4, and RN (i.e., 52, 89, 99, 71, and 83) previously recorded in the electronic database for a product identical to C1. By the same token, when the image data reference values are compared to each other (i.e., when the similarity scores of R1, R2, R3, R4, and RN in the vertical column are correlated against the values of R1, R2, R3, R4 and RN in the horizontal row), it is apparent that RN in the vertical row has the highest average similarity score of 94.25. Thus, given the data set of Table 1, the control circuit 210 of the computing device is programmed to replace the reference data RN associated with the highest similarity score 94.25 from the electronic database 140, and to replace this data with the actual image data obtained from candidate product 190 C1, which will now become the new reference data model for this product for purposes of future scans (until a new, better, reference model is found).
In some aspects, as mentioned above, the computing device 150 is programmed such that, in response to a determination, by the computing device 150, that the similarity score is below the predetermined similarity threshold, the computing device 150 transmits an alert signal to an electronic device of a worker at the retail store, prompting a worker tasked with monitoring the self-checkout terminal 110 to inspect the candidate product 190 that the customer is attempting to check out in the product-scanning area 115 of the self-checkout terminal 110 (i.e., to verify whether the label 195 scanned by the scanner 120 is actually the candidate product 190 with which this label 195 is actually associated, or whether there is ticket-switching (or some other error/anomaly) is involved). Exemplary process flow for the determination, by the computing device 150, of whether or not to generate an alert is described below with reference to
With reference to
If the query data indicates that the product-scanning area 115 is empty, then the control circuit 210 is programmed to not generate an alert (step 472). If the query data indicates that the product-scanning area 115 is not empty, the control circuit 210 is programmed to determine whether the query image data is of sufficiently high quality to make a high confidence determination (step 460). If the query image data is determined by the control circuit 210 to not be of good quality, then the control circuit 210 is programmed to not generate an alert (step 474). If the query image data is determined by the control circuit 210 to be of good quality, then the control circuit 210 is programmed to analyze the verification process and to determine whether the initial determination (of whether ticket-switching is likely or not) can be trusted with a high confidence (step 465). If the control circuit 210 determines that the answer is no, then the control circuit 210 is programmed to not generate an alert (step 476). On the other hand, if the answer is yes, then the control circuit 210 is programmed to generate an alert (step 475), and to halt operation of the self-checkout terminal 110 and prompt manual recognition of the candidate product 190 by a chaperone tasked with monitoring the self-checkout terminal (step 480), and to transmit data indicative of this alert to the electronic database 140 for storage (step 485).
In some aspects, as mentioned above, when the alert is generated by the computing device 150, the control circuit 210 of the computing device 150 is programmed to transmit a signal that causes the self-checkout terminal 110 to halt operations relating to the processing of the purchase of the candidate product 190, pending the manual verification of the candidate product 190 by the chaperone. More specifically, in some embodiments, if the chaperone verifies that the scanned label 195 actually corresponds to the candidate product 190 (i.e., no ticket-switching is involved), then the chaperone can enter, via the chaperone’s electronic device, an input indicating that the chaperone validated the identity of the candidate product 190, and the computing device 150 would then transmit a control signal to the self-checkout terminal 110 in order to permit the self-checkout terminal 110 to process the purchase of the candidate product 190 the identity of which has been validated. On the other hand, if the input by the chaperone indicates that the scanned label 195 does not correspond to the candidate product 190 (i.e., ticket-switching may be involved), then the computing device 150 would transmit a control signal to the self-checkout terminal 110 in order to restrict the self-checkout terminal 110 from processing the purchase of the candidate product 190 the identity of which has been validated. It will be appreciated that the chaperone may input verification data directly into the self-checkout terminal 110 in order to either permit the self-checkout terminal 110 to process the purchase of the candidate product 190, or to restrict the self-checkout terminal 110 from processing the purchase of the candidate product 190.
The method 300 of
As discussed above, the electronic data obtained by the scanner 120 and sensors 130 and 135 is transmitted over the network 125 to an electronic database 140 for storage and is available for retrieval from the electronic database 140 by the computing device 150. To that end, the method 300 of
The method 300 of
After the above-mentioned electronic data is obtained by the computing device 150, the method 300 of
As pointed out above, the correlation step 360 and the determining step 370 enable enables the control circuit 210 of the computing device 150 to determine whether the similarity score of the candidate product 190 is above or below a predetermined similarity threshold that is predetermined to be indicative of ticket-switching. If the similarity score of the candidate product 190 is above a predetermined similarity threshold that is predetermined to be indicative of ticket-switching, the computing device 150 estimates that there is no ticket-switching involved, and the method 300 includes permitting the self-checkout terminal 110 to process a purchase of the candidate product 190 (step 380). On the other hand, if the similarity score of the candidate product 190 is below the predetermined similarity threshold that is predetermined to be indicative of ticket-switching, the computing device 150 estimates that there is ticket-switching involved, and the method 300 includes restricting the self-checkout terminal 110 from processing the purchase of the candidate product 190 (step 390).
As mentioned above, when the computing device 150 determines that the similarity score is below the predetermined similarity threshold, the computing device 150 transmits an alert signal to an electronic device of a worker at the retail store, prompting the worker to inspect the candidate product 190 that the customer is attempting to check out in the product-scanning area 115 of the self-checkout terminal 110 (i.e., to verify whether the label 195 scanned by the scanner 120 is actually the candidate product 190 with which this label 195 is actually associated, or whether there is ticket-switching (or some other error/anomaly) is involved). In some aspects, as mentioned above, in conj unction with generating the alert, the control circuit 210 of the computing device 150 is programmed to transmit a signal that causes the self-checkout terminal 110 to halt operations relating to the processing of the purchase of the candidate product 190, pending the manual verification by the chaperone.
If the chaperone verifies that the scanned label 195 actually corresponds to the candidate product 190 (i.e., no ticket-switching is involved), then the chaperone can enter, via the chaperone’s electronic device, an input indicating that the chaperone validated the identity of the candidate product 190, and the computing device 150 then transmits a control signal to the self-checkout terminal 110 to permit the self-checkout terminal 110 to process the purchase of the candidate product 190 the identity of which has been validated. On the other hand, if the input by the chaperone indicates that the scanned label 195 does not correspond to the candidate product 190 (i.e., ticket-switching may be involved), then the computing device 150 transmits a control signal to the self-checkout terminal 110 to restrict the self-checkout terminal 110 from processing the purchase of the non-valid candidate product 190. In other words, if the computing device 150 generates an alert indicative of possible ticket-switching with respect to a given candidate product 190 and halts operation of the self-checkout terminal 110, the self-checkout terminal 110 chaperone must clear the alert in order for the self-checkout terminal 110 to be able to fully process the purchase transaction of that product.
The systems and methods described herein advantageously detect ticket-switching at self-checkout terminals with a high degree of precision and present an effective mechanism to detect and prevent fraudulent activity at self-checkout terminals, which can serve as a deterrent of fraudulent activity and save the retailers millions/billions of dollars in revenue that is presently lost as a result of fraudulent activities at self-checkout terminals.
Those skilled in the art will recognize that a wide variety of other modifications, alterations, and combinations can also be made with respect to the above-described embodiments without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.
This application claims the benefit of U.S. Provisional Pat. Application No. 63/000,123, filed Mar. 26, 2020, which is incorporated herein by reference in its entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2021/023207 | 3/19/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63000123 | Mar 2020 | US |
