Patent Application
20250218170
The present disclosure relates to item recognition techniques. Physical stores have been implementing item recognition systems to reduce shrink (e.g., theft, loss, incorrect pricing, etc.) of items at self-checkout stations. These systems capture images of items being purchased to identify the items and to determine whether a user is attempting to trick the checkout station into charging for different items than the items the user is purchasing. These systems, however, rely primarily on the size, shape, and color of items to identify the items, which makes it difficult for the system to distinguish between certain classes of items (e.g., regular vs. organic produce, different brands of items, and different quantities of items). As a result, these systems may misidentify items, which results in shrink.
Physical stores have been implementing item recognition systems to reduce shrink (e.g., theft, loss, incorrect pricing, etc.) of items at self-checkout stations. These systems, however, rely primarily on the size, shape, and color of items to identify the items, which makes it difficult for the system to distinguish between certain classes of items (e.g., regular vs. organic produce, different brands of items, and different quantities of items). As a result, these systems may misidentify items, which results in shrink.
The present disclosure describes an item recognition system that uses other information to improve or enhance the item recognition process. When a user attempts to purchase an item, the system captures an image of the item. The system then analyzes the image as well as the user's shopping history to identify the item. For example, the image alone may not allow the system to determine whether the item is a regular or organic item. The system may determine from the user's shopping history that the user tends to purchase organic items rather than regular items. In response, the system may identify the item being purchased as an organic item rather than a regular item.
In certain embodiments, the system provides several technical advantages. For example, the system may improve the accuracy of an item identification or reduce the chances that the system misidentifies an item. By supplementing the image of the item with the shopping history of the user, the system considers additional information, which may improve the accuracy of the item identification. As another example, the system may improve the functioning of a computer system. By considering additional information when identifying the item, the system reduces the chances of misidentifying the item, which typically requires additional input from the user or another associate to correct, consuming additional processor resources and requiring additional electrical power. As a result, the system reduces the amount of processor resources and power consumption, which is an improvement to the functioning of a computer system.
To counter this malicious behavior, the system 100 uses the cameras 104 to capture images of an item as the item is being scanned. The system 100 may then use computer vision techniques to identify the item from the image. The image identification may serve as an extra check on whether the user has scanned the correct barcode. These computer vision techniques, however, are not foolproof. Typically, these techniques rely primarily on shapes, sizes, and colors to distinguish or identify items. Some items, however, share the same shapes, sizes, and/or colors. As a result, these techniques may not determine when a user scanned an incorrect barcode if the items share similar shapes, sizes, and/or colors. For example, it may be difficult for the system to distinguish between two boxed items when the boxes have similar shapes, sizes, and/or colors. As another example, it may be difficult for the system to distinguish between a regular produce item and an organic produce item that looks similar to the regular produce item.
Generally, the system 100 augments the computer vision techniques with other information that helps in identifying an item shown in the image captured by the cameras 104. For example, the system 100 may analyze a user's purchase history to determine the items that a user has purchased or tends to purchase. The system may then use this information to identify an item as an item that the user previously purchased as opposed to an item that the user has never purchased. As another example, the system 100 may analyze other information (e.g., inventory counts in a store or expiration dates of items in a store) to identify an item. In this manner, the system 100 produces a more accurate identification for the item, which may reduce shrink.
The computer system 108 may be integrated with or separate from the checkout system. The computer system 108 may be in communication with the checkout system. Generally, the computer system 108 may assist in identifying items that are scanned by the scanners 102. For example, the computer system 108 may use computer vision techniques as well as other information (e.g., a user's purchase history) to identify an item. As seen in
The processor 110 is any electronic circuitry, including, but not limited to one or a combination of microprocessors, microcontrollers, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), and/or state machines, that communicatively couples to the memory 112 and controls the operation of the computer system 108. The processor 110 may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The processor 110 may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The processor 110 may include other hardware that operates software to control and process information. The processor 110 executes software stored on the memory 112 to perform any of the functions described herein. The processor 110 controls the operation and administration of the computer system 108 by processing information (e.g., information received from the checkout system and memory 112). The processor 110 is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The processor 110 is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set of functions or actions, even if different processing devices perform different functions or actions in the set.
The memory 112 may store, either permanently or temporarily, data, operational software, or other information for the processor 110. The memory 112 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, the memory 112 may include random access memory (RAM), read only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage device or a combination of these devices. The software represents any suitable set of instructions, logic, or code embodied in a computer-readable storage medium. For example, the software may be embodied in the memory 112, a disk, a CD, or a flash drive. In particular embodiments, the software may include an application executable by the processor 110 to perform one or more of the functions described herein. The memory 112 is not limited to a single memory and may encompass multiple memories contained in the same device or computer or distributed across multiple devices or computers. The memory 112 is considered to store a set of data, operational software, or information if the multiple memories collectively store the set of data, operational software, or information, even if different memories store different portions of the data, operational software, or information in the set.
The computer system 108 receives an image 202 of an item. The image 202 may have been captured by one or more cameras 104 in the system 100 when a user was attempting to purchase the item. The image 202 may show the appearance of the item (e.g., the shape, size, or color of the item). The computer system 108 may use the image 202 to identify the item.
The computer system 108 determines one or more potential identities 204 for the item based on the appearance of the item in the image 202. The computer system 108 uses any technique (e.g., computer vision techniques) to determine the potential identities 204 from the image 202. In the example of
The computer system 108 compares each of the probabilities 206 against a threshold 208. The threshold 208 may indicate a minimum probability. If the probability 206 for an identity 204 does not exceed the minimum probability, then the computer system 108 may not consider that identity 204 as a potential identity for the item shown in the image 202. By comparing each of the probabilities 206 against the threshold 208, the computer system 108 may eliminate one or more identities 204 as potential identities for the items shown in the image 202. The computer system 108 may also determine one or more identities 204 that may be potential identities for the item shown in the image 202 because those identities 204 have probabilities 206 that exceed the threshold 208. As an example, if the threshold 208 is set at 60%, then each identity 204 with a probability 206 that does not exceed 60% may be eliminated as potential identifies for the item shown in the image 202. Additionally, each identity 204 with a probability 206 that exceeds 60% may be kept as a potential identity for the item shown in the image 202.
The computer system 108 uses information about the user who is attempting to purchase the item shown in the image 202 to augment the identification of the item. In the example of
The computer system 108 determines one or more weights 212 (e.g., probability weights) using the purchase history 210 for the user. The weights 212 may be determined based on the previous shopping history of the user (e.g., how many times the user had previously purchased certain items or certain types of items). For example, an item may receive a greater weight 212 the more the user has previously purchased the item. An item may receive a lesser weight 212 the fewer times the user has purchased the item. As another example, a class of items (e.g., organic items or a certain brand) may receive a greater weight 212 the more items from that class the user has previously purchased. The class of items may receive a lower weight 212 the fewer items from that class the user has previously purchased. As yet another example, an item may receive a lower weight 212 the more times the user has returned that item after purchasing the item. An item may receive a greater weight 212 the more times the user has kept the item after purchasing the item.
The computer system 108 applies the weights 212 (e.g., probability weights) to the probabilities 206 to produce weighted probabilities 214. In some embodiments, the computer system 108 applies the weights 212 to the probabilities 206 by multiplying the probabilities 206 by the weights 212 to produce the weighted probabilities 214. In the example of
The computer system 108 then compares the weighted probabilities 214 against the threshold 208 to determine an identity 216 for the item shown in the image 202. Using the example of
The computer system 108 begins by retrieving the purchase history 210 for the user. The user may have been identified when the user scanned an identification tag or loyalty tag during the checkout process. The computer system 108 retrieves the purchase history 210 for the identity. As seen in
The computer system 108 determines one or more weights 212 (e.g., probability weights) from the purchase history 210. In the example of
The computer system 108 then determines whether to apply the weights 212 (e.g., probability weights) to certain probabilities 206. For example, the computer system 108 may determine whether the identities 204 match the item 302 for the weight 212. If the identity 204 matches the item 302 for the weight 212, then the computer system 108 may determine that the weight 212 should be applied to the probability 206 for the identity 204. As another example, the computer system 108 may determine characteristics 306 (e.g., organic vs. regular, brand, etc.) for each identity 204. The computer system 108 may determine whether the weight 212 is for items 302 that share a common characteristic 304 that matches the characteristic 306. If the common characteristic 304 matches the characteristic 306, then the computer system 108 may determine that the weight 212 should be applied to the probability 206 for that identity 204. In the example of
Applying the weight 212A to the probability 206A produces the weighted probability 214A. Applying the weight 212B to the probability 206B produces the weighted probability 214B. In this manner, the weighted probabilities 214A and 214B reflect the probabilities 206A and 206B after taking into consideration the purchase history 210 of the user.
As seen in
After determining the identity 216 for the item, the computer system 108 may add the identified item to the transaction, and the computer system 108 may also add the cost of the identified item to the transaction. The user may then pay for the transaction and purchase the item.
The computer system 108 compares multiple weighted probabilities 214 with the threshold 208. In the example of
The computer system 108 may present a prompt 502 to ask the user or a store associate to identify the item. The computer system 108 may present the prompt 502 on the display 106. The prompt 502 may include images 504 for the identities 204. In the example of
The user or store associate may make a selection 506 to select one of the images 504A or 504B in the prompt 502. After making the selection 506, the computer system 108 determines the identity 216 of the item to be the identity 204 shown in the selected image 504A or 504B. The computer system 108 may then add the identified item to the transaction. In this manner, the computer system 108 allows a user or a store associate to identify the item when the weighted probabilities 214 allow for multiple potential identities. The computer system 108 may reduce the number of images 504 included in the prompt 502 by first eliminating identities 204 based on whether their weighted probabilities 214 fall below the threshold 208, which simplifies the identification task for the user or store associate.
The computer system 108 may consider any information when determining the weight 212 (e.g., probability weight) to apply to a probability 206. In the example of
The computer system 108 may consider inventory counts 604 (which may be retrieved from an inventory system of the store) when determining the weight 212 (e.g., probability weight). Each inventory count 604 may indicate a number of a particular item remaining in a store. The computer system 108 may determine a greater weight 212 for an item 302 when that item 302 has a higher inventory count 604. The larger inventory count 604 may indicate a higher probability that a user took that item 302 from the shelf. As a result, the computer system 108 may determine a greater weight 212 for a larger inventory count 604. As another example, a lower inventory count 604 may indicate a lower probability that the user removed the item 302 from the shelf. As a result the computer system 108 may determine a lower weight 212 for a lower inventory count 604. In some embodiments, the computer system 108 determines the opposite (e.g., a greater weight 212 for a lower inventory count 604 (popular item) and a lower weight 212 for a greater inventory count 604 (unpopular item)).
The computer system 108 may consider the expiration date 606 (which may be retrieved from an inventory system of the store) when determining the weight 212 (e.g., probability weight). The computer system 108 may determine (e.g., from the purchase history 210) that the user is less likely to purchase items that are close to their expiration dates. As a result, the computer system 108 may determine a greater weight 212 when the expiration date 606 for an item 302 is further out and a lower weight 212 when the expiration date 606 for an item 302 is close.
The computer system 108 considers the return 608 of items indicated in the purchase history 210 when determining the weight 212 (e.g., probability weight). The return 608 may indicate that the user returned a previously purchased item. The computer system 108 may weigh the return 608 negatively against the weight 212. As a result, the computer system 108 may determine a greater weight 212 when an item 302 has not been returned and a lower weight 212 when the item 302 was previously returned.
In block 702, the computer system 108 determines a first probability 206A and a second probability 206B. The first probability 206A may indicate a likelihood that a user purchased an item with an identity 204A. The second probability 206B indicates a likelihood that the user purchased an item with the identity 204B. The computer system 108 have determined these probabilities 206 by analyzing an image 202 of an item that was being scanned for purchase. For example, the computer system 108 may consider the size, shape, and/or color of the item in the image 202 to determine the probabilities 206 for the identities 204.
In block 704, the computer system 108 determines characteristics 304 of a previous purchase. The computer system 108 may use information in the purchase history 210 of the user to determine the characteristics 304 of the previous purchase. The characteristics 304 may reveal any suitable information about the items 302 in the previous purchase. For example, the characteristics 304 may indicate a brand, color, organic status, etc. of the previous purchase.
The computer system 108 may determine weights 212 (e.g., probability weights) for the items 302 in the previous purchase. The weights may apply to singular items 302 or to multiple items 302 that share common characteristics 304. In block 706, the computer system 108 applies a first weight 212A to the first probability 206A and a second weight 212B to the second probability 206B to produce weighted probabilities 214A and 214B. In some embodiments, the computer system 108 applies the weights 212 by multiplying the probabilities 206 by the corresponding weights 212 to produce the weighted probabilities 214.
In block 708, the computer system 108 assigns the identity 216 to the item. The computer system 108 may determine the identity 216 by comparing the weighted probabilities 214 with the threshold 208. If one of the weighted probabilities 214 exceeds a threshold 208 and the other weighted probabilities 214 fall below the threshold 208, then the computer system 108 may determine the identity 216 to be the identity 204 for the weighted probability 214 that exceeded the threshold 208. If multiple weighted probabilities 214 exceed the threshold 208, then the computer system 108 may present a prompt 502 for the user or a store associate to select the identity 216 for the item.
After the computer system 108 determines the identity 216 of the item, the computer system 108 may compare the determined identity 216 with a scanned identity 802. The scanned identity 802 may be the identity of the item indicated by a scanned bar code. The computer system 108 may determine whether the determined identity 216 matches the scanned identity 802. If the identity 216 and the scanned identity 802 do not match, the computer system 108 communicates an alert 804 indicating the mismatched identity. The alert 804 may signal a store associate to review the purchase to determine whether a misidentification happened. The mismatching identity may be evidence of attempted shrink, which the store associate should resolve. In this manner, the computer system 108 detects attempted shrink, using image recognition techniques and information about the user.
In a first example operation, a user scans a produce item (e.g., using the scanner 102). The cameras 104 capture an image 202 of the produce item. The computer system 108 uses computer vision techniques to analyze the image 202 to determine that the produce item is either a regular banana or an organic banana. The computer system 108 may determine that both the probability 206 that the produce item is a regular banana and the probability 206 that the produce item is an organic banana exceed the threshold 208. The computer system 108 then reviews the purchase history 210 of the user, which shows that the user previously purchased organic bananas rather than regular bananas. In response, the computer system 108 determines a large weight 212 (e.g., probability weight) for organic bananas and a low weight 212 (e.g., probability weight) for regular bananas. The computer system 108 then applies the large weight 212 to the probability 206 for organic bananas and the low weight 212 to the probability 206 for regular bananas. The computer system 108 compares the resulting weighted probabilities 214 to the threshold 208 and determines that the weighted probability 214 for the organic bananas exceeds the threshold 208 and the weighted probability 214 for the regular bananas falls below the threshold 208. In response, the computer system determines that the user is attempting to purchase organic bananas.
In a second example operation, a user scans a boxed item. The cameras 104 capture an image 202 of the boxed item. The computer system 108 uses computer vision techniques to analyze the image 202 to determine that the boxed item is either of a first brand or a second brand. The computer system 108 may determine that both the probability 206 that the boxed item is of the first brand and the probability 206 that boxed item is of the second brand exceed the threshold 208. The computer system 108 then reviews the purchase history 210 of the user, which shows that the user previously purchased items 302 produced by the second brand rather than items 302 produced by the first brand. In response, the computer system 108 determines a large weight 212 (e.g., probability weight) for items 302 of the second brand and a low weight 212 (e.g., probability weight) for items 302 of the first brand. The computer system 108 then applies the large weight 212 to the probability 206 for the first brand and the low weight 212 to the probability 206 for the second brand. The computer system 108 compares the resulting weighted probabilities 214 to the threshold 208 and determines that the weighted probability 214 for the second brand exceeds the threshold 208 and the weighted probability 214 for the first brand falls below the threshold 208. In response, the computer system determines that the user is attempting to purchase an item produced by the second brand.
The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
In the following, reference is made to embodiments presented in this disclosure. However, the scope of the present disclosure is not limited to specific described embodiments. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice contemplated embodiments. Furthermore, although embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the scope of the present disclosure. Thus, the following aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s). Aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.”
The embodiments of the present disclosure may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.
Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
While the foregoing is directed to embodiments of the present disclosure, other and further embodiments may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
